TITLE OF THE INVENTION 

IMAGE SENSING APPARTUS AND METHOD CAPABLE OF MERGING 

FUNCTION FOR OBTAINING HIGH-PRECISION IMAGE BY 
SYNTHESIZING IMAGES AND IMAGE STABILIZATION FUNCTION 

BACKGROUND OF THE INVENTION 

The present invention relates to an image sensing 
apparatus, having an image stabilization function, 
capable of obtaining a high-resolution image by 
slightly shifting an image formation position by an 
optical system, on an image sensing device for 
performing photo-electric conversion on the image, and 
an image synthesis apparatus for synthesizing images 
obtained by the image sensing apparatuses . 

An electronic still camera which uses a solid- 
state image sensing device, such as a charge coupled 
device (CCD), instead of a silver-halide film has been 
commercialized. An electronic still camera is superior 
to a camera using the silver-halide film in 
instantaneity , however, inferior in resolution and 
dynamic range. 

In order to improve the resolution, which is one 
of the above defects of an electronic still camera, an 
image sensing apparatus adopting pixel shifting method 



has been proposed. In the pixel shifting method, a 
plurality of images are sensed while slightly shifting 
the image formation position of an image, incoming 
through an optical system, on an image sensing device 
for performing photo-electric conversion on the image, 
and the plurality of sensed images are synthesized 
using a predetermined method to obtain a single high- 
resolution image. As the prior arts of the pixel 
shifting method, there are following Japanese Patent 
Application Laid-Opens, for example. 

In the Japanese Patent Application Laid-Open No. 
60-27278, a wedge-shaped prism provided in front of the 
lens systems is rotated about the optical axis, thereby 
the light path of an optical image formed on an image 
sensing device is shifted in parallel to the optical 
axis of the lens system. Then, output images are 
synthesized to obtain a single high- resolution image. 

In the Japanese Patent Application Laid-Open No. 
60-91774, in an optical system configured with a 
magnification optical sub- system and a master optical 
system, a part of the lens of the master optical system 
is shifted in the vertical direction with respect to 
the optical axis, thereby the light path of the optical 
image formed on an image sensing device is shifted in 
parallel to an optical axis of the optical system. 



Then the obtained output images are synthesized to 
obtain a single high-resolution image * 

Further, in the Japanese Patent Application Laid- 
Open No. 61-236282, a transparent plane parallel plate 
provided in front of an image sensing device is rotated 
about an axis which is perpendicular to the optical 
axis of the image sensing device, thereby the light 
path of the optical image formed on the image sensing 
device is shifted in parallel to the optical axis. 
Then the output images are synthesized to obtain a 
single high-resolution image. 

In the Japanese Patent Application Laid-Open No. 
7-287268 (U.S. Patent Application No. 08/339,407), a 
variable apical angle prism provided in front of an 
optical system is operated on the basis of a vibration 
signal and a pixel shifting signal, thereby shifting 
the light path of the optical image formed on an image 
sensing device in parallel to the optical axis. 
Accordingly, compensation of vibration caused by a user 
as well as improvement of resolution of an image are 
achieved simultaneously. 

Note, in this specification, any unintentional 
undesired movement of camera is expressed as 
"vibration", and the movement is not limited to 
periodic motion. 



However, in the aforesaid pixel shifting methods, 
it takes a long time from the first image signal until 
the last image signal are obtained, similarly to the 
case of performing multiple exposure in a still camera. 
Thus, when the electric still camera is vibrated, the 
quality of an image decreases, namely, a high- 
resolution image may not be obtained by performing 
pixel shifting. Thus, in order to overcome this 
problem, there are following Japanese Patent 
Application Laid-Opens, for example. 

In the Japanese Patent Application Laid-Open No. 
7-240932 (U.S. Patent Application No. 08/391,388), 
using a variable apical angle prism provided in front 
of the optical system or a moving lens system provided 
behind the optical system, both compensation of 
vibration and improvement of resolution of the image 
are achieved at the same time. 

Further, according to the Japanese Patent 
Application Laid-Open No. 7-287268 (U.S. Patent 
Application No. 08/339,407), because resolution in 
pixel shifting operation decreases when the focal 
length of the optical system is larger than a 
predetermined value, pixel shifting is disabled in such 
a case. 

Further, in order to widens dynamic range, which 
is the other defect of the electronic still camera. 



-4- 



there are the following Japanese Patent Application 
Laid-Opens . 

In the Japanese Patent Application Laid-Open No. 
1-319370, an image sensing device is exposed a 
5 plurality of times with different luminous exposures, 
and a plurality of images obtained under this operation 
are synthesized to form a signal image of wide dynamic 
range . 

In the Japanese Patent Application Laid-Open No. 
10 7-264488, a plurality of image sensing devices having 
different sensibilities are used, and a plurality of 
images obtained by these image sensing devices are 
synthesized to form a single image of wide dynamic 
range . 

15 Furthermore, as a technique for overcoming the 

aforesaid two problems at the same time, the Japanese 
Patent Application Laid-Open No. 8-37628 (U.S. Patent 
Application No. 08/505,608) discloses that at least one 
of a plurality of images obtained while performing 

20 pixel shifting is sensed in different luminous exposure 
from luminous exposure used for sensing other images, 
thereby obtaining an image of high resolution and wide 
dynamic range. 

Further, in an image sensing apparatus having an 

25 image sensing device, it is possible to determine 

vibration of an image in advance to actually sensing 
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the image by obtaining a movement vector of the image 
from time -sequential outputs from the image sensing 
device. Accordingly, in the Japanese Patent 
Application Laid-Open No. 2-57078 as a prior art in 
this field, a movement vector of an image is detected 
continuously, and, when the movement vector becomes the 
smallest, the image sensed at that time is decided as 
an image to be recorded, thereby reducing the effect of 
the vibration of an image sensing apparatus during 
exposure. 

Furthermore, in the Japanese Patent Application 
Laid-Open No. 8-172568, movement vectors between a 
plurality of images sensed while performing pixel 
shifting are obtained, and component of vibration due 
to vibration of an image sensing apparatus or of an 
object are removed by performing interpolation, 
thereafter, the images are synthesized to form a single 
image of high resolution. 

However, in the aforesaid conventional examples 
disclosed in the Japanese Patent Application Laid-Opens, 
there are following defeats . 

In the methods disclosed in the Japanese Patent 
Application Laid-Open Nos . 60-27278, 60-91774, and 61- 
236282, no vibration correction mechanism is provided. 
Therefore, it is not possible to obtain a high- 
resolution image when blurring of an image caused by 



vibration of an image sensing apparatus is large. This 
is because the operation for obtaining a plurality of 
images in the pixel shifting method is the same as that 
of multiple exposure, as described above, and the time 
required in these operations, namely the time when the 
first image is sensed until the last image is sensed, 
is longer than the time required for performing a 
normal image sensing operation. As a result, effects 
of vibration on an image is greater when performing 
pixel shifting operation than when performing the 
normal image sensing operation, vibration 

Further, according to the Japanese Patent 
Application Laid-Open No. 7-287268 (U.S. Patent 
Application No. 08/339,407), the variable apical angle 
prism, which is the light path shifting means used for 
vibration compensation and pixel shifting operation, is 
provided in front of the optical system, therefore, the 
coefficient (vibration compensation coefficient) for 
converting a vibration signal into a value for 
operating the variable apical angle prism does not 
change in response to zooming operation. Accordingly, 
when performing zooming operation, only the coefficient 
(pixel shifting coefficient) for converting a pixel 
shifting signal into a value for operating the variable 
apical angle prism needs to be changed. However, when 
the light path shifting means is provided in the middle 



of the optical system in order to down-sizing the 
optical system, it is necessary to convert both the 
vibration signal and the pixel shifting signal into 
values using specific coefficients when performing 
zooming operation, and operate the light path shifting 
means in accordance with the values, but there is no 
disclosure on such the conversion and operation in the 
Japanese Patent Application Laid-Open No. 7-287268 (U.S. 
Patent Application No. 08/339,407). 

Furthermore, in the Japanese Patent Application 
Laid-Open No. 7-287268 (U.S. Patent Application No. 
08/339,407), a pixel shifting mechanism is applied to a 
video camera (camcorder) for recording a moving image, 
therefore, the interval for taking images is fixed to 
the field frequency of a moving image. However, when 
the pixel shifting mechanism is applied to a so-called 
electronic still camera for recording a still image, it 
is advantageous to use an image sensing device whose 
image-taking interval can be changed on the basis of 
the charging period of the image sensing device (i.e., 
luminance of an object), since the camera can sense an 
object in a wide luminance range. In this case, as the 
image- taking interval becomes longer, effects of 
vibration on the image becomes stronger, therefore, 
delicate control of the pixel shifting operation in 
accordance with image sensing conditions is required. 



However, in the Japanese Patent Application Laid-Open 
No, 7-287268 (U.S. Patent Application No. 08/339,407), 
only change in pixel shifting operation in accordance 
with a focal length of the optical system is disclosed. 
In addition, there is no teaching on dynamic range 
expansion. 

In the Japanese Patent Application Laid-Open No. 
7-240932 (U.S. Patent Application No. 08/391,388), 
pixel shifting is performed even when the resolution of 
vibration compensation is not good or vibration 
compensation has failed. As a result, a high- 
resolution image is not obtained; on the contrary, the 
quality of the image obtained by performing pixel 
shifting operation would be lower than an image 
obtained without performing pixel shifting operation. 

According to the Japanese Patent Application Laid- 
Open No. 7-287268 (U.S. Patent Application No. 
08/339,407), under conditions in which it is predicted 
that resolution higher than a predetermined level can 
not be obtained by performing pixel shifting operation, 
the pixel shifting operation is disabled. However, the 
prediction is not performed on the basis of an actual 
effect of vibration on an image. Therefore, similarly 
to the cases of other references as explained above, 
when an image sensing apparatus vibrates by a large 
displacement amount, the obtained image would have a 



lower quality than an image obtained without performing 
pixel shifting operation. 

Further, according to the Japanese Patent 
Application Laid-Open No. 60-91774, resolution of an 
image is increased, however, there is no teaching on 
dynamic range expansion. 

On the contrary, in the Japanese Patent 
Application Laid-Open Nos. 1-319370 and 7-264488, 
dynamic range expansion is explained, however, how to 
increase resolution of an image is not discussed. 

Whereas, in the Japanese Patent Application Laid- 
Open No. 8-37628 (U.S. Patent Application No. 
08/505,608), methods for increasing resolution of an 
image and widening dynamic range are disclosed, however, 
there is no detailed description on method for 
determining luminous exposures to be used for sensing a 
plurality of images for dynamic range expansion. 
Therefore, the disclosed method is not possible to 
delicately cope with various scenes which have various 
luminous distributions. Further, pixel shifting 
operation requires longer time for performing exposing 
operation, similarly to a case of performing multiple 
exposure operation, as described above, therefore, it 
is necessary to cope with vibration problem. However, 
there is no teaching on any technique for overcoming 
the vibration problem. 
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Further, in the Japanese Patent Application Laid- 
Open No. 2-57078, there is no teaching on pixel 
shifting operation, therefore, only reduction of effect 
of vibration on an image sensed in a normal image 
sensing operation is achieved. Thus, improvement in 
resolution of an image is not expected. Furthermore, 
blurring due to a movement of an object can not be 
reduced. 

Further, in a case of performing pixel shifting 
operation to increase resolution of an image, it is 
necessary to control the shift amount to be a 
predetermined amount based on the interval between 
pixels. However, there is no teaching on optical 
vibration compensation means using, e.g., a variable 
apical angle prism in the Japanese Patent Application 
Laid-Open No. 8-172568, therefore, blurring of an image 
while performing pixel shifting operation is large and 
occurs at random. Therefore, there is no guarantee 
that pixel shifting by the predetermined shift amount 
is always performed. In addition, even though blurring 
of an object is corrected by performing interpolation, 
possibility of obtaining a high resolution image is low. 
Furthermore, there is no teaching on warning a user of 
an image not being obtained in desired resolution, nor 
about an alternative suggestion to be followed for 
improving resolution of the image. 
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SUMMARY OF THE INVENTION 



The present invention has been made in 
consideration of the above situation, and has as its 
object to provide an image sensing apparatus and method 
capable of performing image stabilization and pixel 
shifting simultaneously and at high resolution 
regardless of the configuration of the optical system 
and the configuration of image stabilization function. 

According to the present invention, the foregoing 
object is attained by providing an image sensing 
apparatus comprising: image sensing means for 
converting an optical image into electric signals and 
outputting the electric signals as image signals; 
shifting means for shifting an image formation position 
of the optical image formed on the image sensing means 
to a plurality of different positions; first vibration 
detection means for detecting vibration of the image 
sensing apparatus and outputting vibration information; 
control means for controlling the shifting means on the 
basis of the vibration information outputted by the 
first vibration detection means; and image signal 
synthesis means for synthesizing image signals of a 
plurality of images outputted by the image sensing 
means to generate a single image, wherein the image 
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sensing means converts the optical image into electric 
signals at each of the plurality of different image 
formation positions shifted by the shifting means. 

It is another object of the present invention to 
provide an image sensing apparatus capable of reducing 
the effect of movement of an object while performing 
pixel shifting so as to obtain an image of high 
resolution. 

According to the present invention, the foregoing 
object is attained by providing an image sensing 
apparatus comprising: image sensing means for 
converting an optical image into electric signals and 
outputting the electric signals as image signals; image 
sensing control means for controlling the image sensing 
means to sense a plurality of images within a 
predetermined period of time; image signal synthesis 
means for synthesizing image signals of the plurality 
of images outputted by the image sensing means to 
generate a single image; division means for dividing an 
image into a plurality of small areas; relationship 
determination means for determining relationship 
between the plurality of images by each of the 
plurality of small areas; and image synthesis control 
means for controlling image synthesis operation by the 
image signal synthesis means on the basis of the 
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relationship between the plurality of images 
determined by the relationship determination means. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof, 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate embodiments of the invention and, together 
with the description, serve to explain the principles 
of the invention. 

Fig. 1 is a block diagram illustrating a 
configuration of an image sensing apparatus according 
to an embodiment of the present invention; 

Figs. 2A and 2B are diagrams illustrating an 
example of an arrangement of lenses in a lens system 
according to the embodiment of the present invention; 

Figs. 3A and 3B are diagrams for explaining shifts 
of light paths by the lens system according to the 
embodiment of the present invention; 



-14- 



Fig. 4 is an explanatory view of a principle of 
pixel shifting; 

Fig. 5 is a block diagram illustrating a detailed 
configuration of microcomputers of a camera main body 
and a lens and their peripheral units according to the 
embodiment of the present invention; 

Fig. 6 is a flowchart of control processing in the 
camera main body according to first to third 
embodiments of the present invention; 

Fig. 7 is a flowchart of control processing in the 
lens according to the first, fifth, eighth and eleventh 
embodiments of the present invention; 

Fig. 8 is a timing chart of the control processing 
according to the first and eighth embodiments of the 
present invention ; 

Figs. 9A and 9B are drawings for explaining pixel 
shifting method according to the first, fifth and 
eighth embodiments of the present invention; 

Figs. 10A to 10E are drawings for explaining image 
synthesis method according to the first, fifth, eighth 
and eleventh embodiments of the present invention; 

Fig. 11 is a flowchart of control processing in 
the camera main body according to a second embodiment 
of the present invention; 
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Figs. 12A and 12B are views for explaining pixel 
shifting method according to the second and sixth 
embodiments of the present invention; 

Fig. 13A to 13C are drawings for explaining image 
synthesis method according to the second and sixth 
embodiments of the present invention ; 

Fig. 14 is a flowchart of control processing in 
the camera main body according to a third embodiment of 
the present invention; 

Fig. 15 is a block diagram illustrating a detailed 
configuration of microcomputers of the camera main body 
and the lens and their peripheral units according to a 
fourth embodiment of the present invention; 

Fig. 16 is a flowchart of control processing in 
the camera main body according to the fourth embodiment 
of the present invention; 

Fig. 17 is a flowchart of control processing in 
the lens according to the fourth embodiment of the 
present invention; 

Fig. 18 is a flowchart of control processing in 
the camera main body according to fifth to eighth, and 
tenth embodiments of the present invention ; 

Fig. 19 is a flowchart showing processes for 
setting luminous exposure performed in processing shown 
in Fig. 18; 
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Fig- 20 is a graph showing characteristics of a 
film and an image sensing device; 

Fig. 21 is a view showing a detection area where 
focus state detection and the photometry are performed; 

Fig. 22 is a timing chart of the control 
processing according to the fifth embodiments of the 
present invention; 

Fig. 23 is a graph showing a feature of the fifth 
embodiment ; 

Fig. 24 is a flowchart of control processing in 
the camera main body according to a sixth embodiment of 
the present invention; 

Fig. 25 is a flowchart of control processing in 
the camera main body according to a seventh embodiment 
of the present invention; 

Figs. 26A and 26B are graphs for explaining a 
principle for calculating a relative shifted amount of 
two images; 

Figs. 27A and 27B are graphs for explaining the 
principle for calculating a relative shifted amount of 
the two images ; 

Fig. 28 is a graph showing relationship between 
shift amount and correlation; 

Fig. 29 is an explanatory drawing of a principle 
of pixel shifting according to an eighth embodiment; 
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Fig. 30 is a graph showing correlation between 
first and second image signals; 

Fig. 31 is an explanatory view showing a case 
where large vibration was occurred while performing 
pixel shifting operation according to an eighth 
embodiment ; 

Fig. 32 is a graph showing correlation between 
first and second image signals; 

Fig. 33 is a flowchart of control processing 
performed in the processing shown in Fig. 18; 

Fig. 34 is an explanatory view showing a track of 
a given point of an image formed on an image sensing 
device according to a ninth embodiment of the present 
invention; 

Fig. 35 is a flowchart of control processing in 
the camera main body according to the ninth embodiment 
of the present invention; 

Fig. 36 is a flowchart of control processing in 
the camera main body according to the ninth embodiment 
of the present invention; 

Fig. 37 is an explanatory view showing a track of 
a given point of an image on an image sensing device 
according to a tenth embodiment of the present 
invention; 

Fig. 38 is a graph showing correlation between 
first and second image signals; 
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Fig. 39 is a flowchart of control processing in 
the camera main body according to the tenth embodiment 
of the present invention; 

Fig. 40 is a flowchart of control processing in 
the camera main body according to an eleventh 
embodiment of the present invention; 

Fig. 41 is an explanatory view showing an image of 
an object formed on a photo -sensing surface of an image 
sensing device; 

Fig. 42 is an explanatory view showing a 
synthesized image of an object; 

Fig. 43 is an explanatory view of divided areas of 
a photo- sensing surface of the image sensing device; 

Fig. 44 is an explanatory view showing a 
determined result of blurring due to movement of the 
object; 

Fig. 45 shows an image obtained by synthesizing a 
plurality of images according to the eleventh 
embodiment of the present invention; 

Fig. 46 is a flowchart of blurring determination 
processing due to vibration performed in the processing 
shown in Fig. 40; 

Fig. 47 is a flowchart of blurring determination 
processing due to movement of the object performed in 
the processing shown in Fig. 40; 
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Fig. 48 is a flowchart of image synthesis 
processing according to the eleventh embodiment; 

Fig, 49 is a view for explaining image synthesis 
method according to a twelfth embodiment; 

Fig, 50 is a view for explaining image synthesis 
method according to the twelfth embodiment; 

Fig. 51 is a view for explaining image synthesis 
method according to the twelfth embodiment; 

Fig. 52 is a flowchart of control processing in 
the camera main body according to the twelfth 
embodiment of the present invention; 

Fig. 53 is a view for explaining image synthesis 
method according to a thirteenth embodiment; 

Fig. 54 is a view for explaining image synthesis 
method according to the thirteenth embodiment; 

Fig. 55 is a flowchart of control processing in 
the camera main body according to the thirteenth 
embodiment of the present invention; and 

Fig. 56 is a flowchart of control processing in 
the camera main body according to a fourteenth 
embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
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Preferred embodiments of the present invention 
will be described in detail in accordance with the 
accompanying drawings. 

As embodiments of the present invention, first to 
fourteenth embodiments are explained below. First, a 
configuration of a camera commonly used in the first to 
fourteenth embodiments is explained below. 
<Conf iguration of Camera) 
Figs. 1 to 5 are common for the first to 
fourteenth embodiments. 

Fig. 1 is a block diagram illustrating a 
configuration of an image sensing apparatus according 
to an embodiment of the present invention. The detail 
will be explained later. 

Figs. 2A and 2B are diagrams illustrating an 
example of an arrangement of lenses in a lens system. 
The lens system is a zoom lens system, and its focal 
length ranges between 10mm to 30mm, namely, third-power 
lens system. Especially, Fig. 2A shows the lenses in 
the wide-angle position (focal length = 10mm), and Fig. 
2B shows the lenses in the telephoto position (focal 
length = 30mm) . 

This lens system is basically configured with four 
groups of lenses: when changing magnification, the 
fourth lens group is stationary, and the first, second 
and third lens groups move; further, when focusing, the 
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second, third and fourth lens groups are stationary, 
and the first lens group moves. Pixel shifting and 
vibration compensation are performed by shifting the 
image formation position of an image on the imaging 
5 surface by shifting the second lens group in the 

vertical direction with respect to the optical axis of 
the lens system. 

Next, an effect of the second lens group on 
shifting a light path is explained with reference to 

10 Figs . 3A and 3B . Figs . 3A and 3B are diagrams 

conceptually showing the four lens groups shown in Figs. 
2 A and 2B. Fig. 3A shows a shift of the light path in 
the lens groups in the image space when the second lens 
group is shifted downward by a distance dL, and Fig. 3B 

15 shows the shift of the light path in the object space 
when the second lens group is shifted downward by the 
distance of dL. 

First, Fig. 3A is explained. A light path of a 
ray incoming from the object space and traveling on the 

20 optical axis of the first lens group is shifted upward 
by the second lens group which is shifted downward, 
passes through the third and fourth lens groups and 
incidents on the imaging surface at a position which is 
above the optical axis of the first, third and fourth 

25 lens groups by the distance of d IM . If a ratio of the 
shifted distance of the second lens group, d L , to the 
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maximum shifted distance on the imaging surface, d IM , 
is defined as decentering sensitivity, S d , then these 
three values have relationship expresses as, 

d IM = S d x d L . . . . (1) 

The decentering sensitivity S d changes in 
accordance with the arrangement of the second to fourth 
lens groups, which means the decentering sensitivity Sd 
changes in accordance with zooming operation according 
to the embodiment of the present invention. Further, 
since a front focusing using the first lens group is 
adopted in the embodiment of the present invention, the 
decentering sensitivity S d does not change in response 
to focusing operation. However, when a rear focusing 
using the fourth lens group is adopted, the decentering 
sensitivity S d changes in response to focusing 
operation. Therefore, the decentering sensitivity S d 
is generally expressed as a function of the focal 
length f and a distance to an object R, namely, S d (f , 
R), thus, the equation (1) may be modified to, 

d IM = s d (f , R) x d L . . „ ( 2 ) 

Next, a shift amount of the second lens group when 
performing pixel shifting operation is explained. Fig. 
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4 is an explanatory view of a principle of pixel 
shifting and shows a magnified view of a photo- sensing 
surface of the image sensing device. Referring to Fig. 
4, photo-sensing elements having a square shape, namely, 
pixels, are regularly arranged on the photo-sensing 
surface at intervals of W Y in the horizontal direction 
and W p in the vertical direction. The resolution of an 
image formed on the photo -sensing surface is determined 
by the intervals W Y and W p between the pixels. However, 
it is possible to increase the resolution of the image 
by taking a plurality of images while shifting relative 
positions between the pixels and the image, and 
synthesizing the images by following a predetermined 
rule to make a single image. For example, when a given 
object point of an image is formed at the center of the 
photo-sensing element, IM1 , image signals of the image 
(entire image signals obtained from the area sensor) 
are read and stored as a first set of image signals, 
next, the image is shifted so that the same object 
point of the image is shifted to a position IM2 on the 
photo -sensing surface, namely shifted to right by X Y = 
W Y /2, and there, a second set of image signals are read 
and stored. Similarly, while shifting the object point 
of the image to positions IM3 and IM4 on the photo- 
sensing surface, third and fourth sets of image signals 
are taken. Thereafter, the four sets of image signals. 
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which provide four times more information on the image, 
are synthesized. Accordingly, spatial resolution of 
the image is doubled both in the horizontal and 
vertical directions . 

In order to shift the image by X Y (=W Y /2) and/or X p 
(=W p /2) in the pixel shifting operation, the effect of 
the second lens group on shifting the light path shown 
in Fig. 3A is utilized. More specifically, in order to 
shift the image upward by X p , the second lens group is 
to be shifted downward by a distance of d L which is 
determined by, 

d L - X p /S d (f , R) . . .(3) 

in accordance with the equation (2). The shift amount 
X p of the image in the pixel shifting operation is 
fixed, however, the decentering sensitivity Sd(f , R) 
changes in accordance with zooming and focusing 
operations. Accordingly, the shift amount of the 
second lens group d L needs to be changed in accordance 
with the state of the lens system. In the present 
invention as described above, data on the decentering 
sensitivity Sd(f , R) corresponding to states of the 
lens system, such as zooming and focusing, is stored as 
a first coefficient in ROM of a microprocessor (CPU) . 
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Next, Fig, 3B is explained. A ray traveling from 
the center of an image along an optical axis C of the 
lens system to the left in Fig. 3B passes through the 
fourth and third lens groups, then shifted upward by 
5 the second lens group which is shifted downward. Then, 
the ray which has passed through the first lens group 
is projected on the object tilted by an angle of 0 OB 
with respect to an axis C which is parallel to the 
optical axis C. If the ratio of the shifted amount of 
10 the second lens group d L to the tilted angle 8 OB with 
respect to the optical axis is defined as the angle 
sensitivity, Se, then these three values have 
relationship expressed as, 

15 6 0B = S e x d L . . .(4) 

The angle sensitivity Se changes depending upon an 
arrangement of lenses in the upstream of the second 
lens group. Namely, in the embodiments of the present 

20 invention, the angle sensitivity Se changes in response 
to zooming and focusing operations . Since the angle 
sensitivity Se is expressed as a function of the focal 
length f and the distance to an object R, namely, Se(f , 
R) , similarly to the decentering sensitivity S d , the 

25 equation (4) may be modified to, 
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6 OB = S 0 (f , R) x d L . 



.(5) 



Next, a shift amount of the second lens group for 
compensating vibration is explained. Assuming that a 
camera having the lens system and the image sensing 
device vibrates in the angular direction so that the 
optical axis of the lens system turns downward by an 
angle of 9^^, blurring of the image due to the 
angular vibration this time corresponds to a case where 
an object is shifted upward with respect to the camera 
by an angle of 6 OB (=9^^). Referring to Fig. 3B, it is 
possible to compensate the shift of the object by 
shifting the second lens group by the direction of d T 
when the object is shifted upward by the angle of 0 OB . 
Therefore, the blurring of the image due to the angular 
vibration can be compensated by shifting the second 
lens group downward by the distance d L calculated on 
the basis of the following equation (6) on the basis of 
the shifted angle, 0 CAMERA , detected by a vibration 
sensor, and the equation (5), 

d L = 9 CAMERA /S e (f, R) ..-(6) 

Since the vibration angle 9^^ changes with 
respect to time and the angle sensitivity S 6 (f , R) also 
changes in response to zooming and focus operations. 



-27- 



* T 



the shift amount of the second lens group needs to be 
changed in accordance with the state of the lens system. 
Therefore, in the present invention, the angle 
sensitivity S 0 (f , R) which changes in response to 
5 zooming and focusing operations is also stored in 

connection with the states of the lens system in the 
ROM of the CPU as a second coefficient, similarly to 
the decentering sensitivity Sd(f, R) . 

Fig, 1 is the configuration of the image sensing 
10 apparatus according to the embodiment of the present 
invention. In Fig. 1 # reference CMR denotes a camera 
main body; and LNS, a lens configured as an 
interchangeable lens, detachable from the camera main 
body CMR. 

15 An explanation of the configuration of the camera 

main body CMR follows. 

Reference CCPU denotes a one-chip microcomputer of 
the camera main body CMR, having ROM, RAM, and analog- 
digital and digital- analog conversion functions. The 

20 microcomputer CCPU performs a series of processes, such 
as automatic exposure ( AE) control, automatic focusing 
(AF) control, and pixel shifting control, by executing 
a sequential program, for the camera, stored in the ROM. 
Thus, the microcomputer CCPU communicates with 

25 respective circuits of the camera main body CMR and the 
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lens LNS in order to control the circuits and the lens 
LNS. 

On a mount unit for connecting the camera main 
body CMR and the lens LNS, four pairs of connection 
terminals are provided. An internal battery BAT in the 
camera main body CMR provides electric power to the 
respective circuits in the camera main body CMR and to 
an actuator, as well as to the lens LNS via a line VCC. 

Reference DCL denotes a signal line for 
transmitting a signal from the microcomputer CCPU of 
the camera main body CMR to a microcomputer LCPU (will 
be explained later) of the lens LNS, and DLC denotes a 
signal line for transmitting a signal from the 
microcomputer LCPU of the lens LNS to the microcomputer 
CCPU of the camera main body CMR, Further, the camera 
main body CMR and the lens LNS are both grounded via a 
line GND . 

Reference IMS denotes an image sensing device, 
such as CCD, and reference IMDR denotes a driver for 
controlling charging of the image sensing device IMS 
and transference of the stored charges in the image 
sensing device IMS. 

Further, reference MEM denotes a memory for 
recording/storing image data of a sensed image, and 
realized by a semi-conductor memory, a magnetic disk, 
and an optical disk, for instance; DISP, a display, 

-29- 



such as a liquid crystal display, for displaying an 
image obtained by the image sensing device IMS as well 
as operation state of the camera; and BS, a beam 
splitter, configured with a half mirror, for leading a 
5 part of the luminous flux of an image to a sensor SNS. 
The sensor SNS has a focus state detection sensor for 
detecting the focus state of the lens system and a 
photometric sensor for detecting luminosity of the 
object . 

10 Reference CNC denotes a connector for connecting 

to an external device, such as a desk- top computer, and 
the connector CNC is used for transmitting the contents 
of the memory MEM to the external device, and 
controlling the camera main body CMR from the external 

15 device using a signal from the external device. 

Reference SWMN denotes a main switch, and when the 
main switch SWMN is turned on, then the microcomputer 
CCPU starts executing a predetermined program relating 
to image sensing operation. 

20 SW1 and SW2 denote switches which operate in 

response to the operation of the release button, and 
SW1 is turned on when the release button is pressed 
halfway (half stroke) and the SW2 is turned on when 
full stroke is made. 

25 SWSF denotes a pixel shifting mode selection 

switch which is used for selecting either permission or 
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prohibition of pixel shifting operation, as well as 
selecting one of a plurality of predetermined pixel 
shifting modes. 

SWIS denotes an image stabilization (IS) selection 
switch for selecting either permission or prohibition 
of image stabilization, 

SWMOD denotes an image sensing mode selection 
switch, and when a user selects one of a predetermined 
image sensing modes, AE mode, AF mode, pixel shifting 
mode, and IS mode, corresponding to the selected image 
sensing mode, are automatically set. 

Next, the configuration of the lens LNS is 
explained. 

Reference LCPU denotes the microcomputer of the 
lens LNS, and configured as a one-chip microcomputer 
having ROM, RAM, and analog- digital and digital-analog 
conversion functions, similarly to the microcomputer 
CCPU of the camera main body CMR. The microcomputer 
LCPU controls a focusing actuator, a zooming actuator, 
a iris diaphragm actuator, and an IS actuator, all of 
which will be explained later, in accordance with 
instructions transmitted from the microcomputer CCPU 
via the signal line DCL. Further, the microcomputer 
LCPU transmits operation state of the lens LNS and 
parameters which are specific to the lens to the 
microcomputer CCPU via the signal line DLC. 
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Reference LI to L4 denote lens groups 
corresponding to the first to fourth lens groups, 
respectively explained with reference to Fig. 2, which 
configure a zoom lens system, and an image of an object 
is formed on the image sensing device IMS via the zoom 
lens system. 

FACT denotes the focusing actuator for moving the 
first lens group LI in the back and forth directions 
along the optical axis to perform focus control. A 
focus encoder FENC detects the position of the first 
lens group LI, which corresponds to information on the 
distance to the object, then the obtained information 
is transmitted to the microcomputer LCPU. 

Reference ZACT denotes the zooming actuator which 
performs zooming operation by moving the first to third 
lens groups LI to L3 in the back and forth directions 
along the optical axis. A zoom encoder ZENC detects 
information on zooming operation, namely, the focal 
length of the zoom lens system, then transmits the 
information to the microcomputer LCPU. 

DFM denotes an iris diaphragm, and DACT denotes an 
iris diaphragm actuator for driving the iris diaphragm 
DFM. 

Further references GRP and GRY are vibration 
sensors, such as vibration -type gyroscopes, and two 
sensors of the same type are provided as the vibration 
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sensors GRP and GRY for sensing the angular vibration 
in the vertical direction (pitch) and horizontal 
direction (yaw) of the camera. The detected results of 
vibration are sent to the microcomputer LCPU. 

The second lens group L2 is provided so as to be 
movable individually in the plane perpendicular to the 
optical axis in the two-dimensional directions. The 
second lens group L2 is driven by a pitch actuator 
IACTP in the vertical direction with respect to the 
optical axis, i.e., in the direction for compensating 
pitch, and driven by a yaw actuator IACTY in the 
horizontal direction (in Fig. 1, the direction normal 
to the paper), namely, in the direction for 
compensating yaw. Note, the shift function is 
disclosed in the Japanese Patent Application Laid-Open 
6-3727 by the same applicant. 

Fig. 5 is a block diagram illustrating a detailed 
configuration of the microcomputers CCPU and LCPU of 
the camera main body CMR and the lens LNS and their 
peripheral units according to the embodiment of the 
present invention. Upper blocks enclosed by a two-dot- 
dash line are included in the microcomputer CCPU of the 
camera main body CMR, and lower blocks enclosed by 
another two -dot -dash line is included in the 
microcomputer LCPU of the lens LNS. 
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Reference numeral 11 denotes an image sensing 
condition setting circuit for setting operation modes, 
such as AE mode, AF mode, pixel shifting mode and IS 
mode; 12 , a timing pulse generator for generating a 
trigger signal for controlling timing of operation of 
the lens system for pixel shifting operation and 
controlling timing for taking image signals from the 
image sensing device; 13, an image sensing device 
operation circuit for taking image signals at a 
predetermined timing under a predetermined charging 
condition in response to control signals generated by 
the image sensing condition setting circuit 11 and the 
timing pulse generator 12; 14, a temporary storage 
circuit for temporarily storing the obtained image 
signals until time for performing synthesis operation; 
15, an image synthesis circuit for synthesizing plural 
sets of image signals obtained while performing pixel 
shifting operation to generate a single image of high 
resolution; and 16, a recording unit, corresponding to 
the memory MEM in Fig. 1, for recording the synthesized 
high-resolution image. 

Further, reference numeral 21 denotes a pixel 
shifting signal generator for generating instruction 
signals (signals having reference waveforms shown in 
Fig. 8. Will be described later in detail) for 
shifting an image formation position for pixel 
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shifting; and 22, a first coefficient generator for 
reading data corresponding to the decentering 
sensitivity Sd(f, R) , which is explained above, from 
the ROM of the microcomputer LCPU in accordance with 
5 the focus and zoom information of the lens system, and 
calculating shift amount instruction values for the 
second lens group L2, so that the amounts of shifts of 
the image formation position in the yaw and pitch 
directions become X Y and X p , respectively. The shift 

10 amount instruction values can be obtained by 

multiplying the amplitude of the instruction signals 
generated by the pixel shifting signal generator 21 by 
the read data (decentering sensitivity Sd(f, R) ) . 

Reference numeral 31 denotes a vibration sensor 

15 which corresponds to the vibration -type gyro GRP and 
GRY, explained above; 32, a vibration signal 
calculation circuit for performing filtering and 
accumulation on an angular velocity signal of the 
vibration detected by the vibration sensor 31, and 

20 calculating a vibration angle; and 33, a second 

coefficient generator for reading data corresponding to 
the angle sensitivity Se(f, R) , explained above, from 
the ROM of the microcomputer LCPU in accordance with 
the focus and zoom information of the lens system, 

25 correcting the value of the vibration angle calculated 
by the vibration signal calculation circuit 32, and 
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calculating shift amount instruction values 
(amplitudes of vibration compensation signals) for the 
second lens group L2 for image stabilization. 

Further, reference numeral 41 denotes a synthesis 
circuit for adding the shift amount instruction value 
for the second lens group L2 for pixel shifting 
calculated by the first coefficient generator 22 and 
the shift amount instruction value for the second lens 
group L2 for image stabilization calculated by the 
second coefficient generator 33; 42, an image 
stabilization (IS) actuator controller for controlling 
the pitch actuator IACTP and the yaw actuator IACTY in 
Fig. 1, so that the second lens group L2 moves in 
accordance with the value obtained by the synthesis 
circuit 41; and 43, a block indicating that the second 
lens group L2 is shifted, in other words, the image 
formation position on the image sensing device 13 is 
shifted. 

< First Embodiment > 

Figs. 6 and 7 are flowcharts of control processing 
by the microcomputer CCPU of the camera main body CMR 
and the microcomputer LCPU of the lens LNS, 
respectively, according to a first embodiment of the 
present invention. 
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First, a flow of the control processing by the 
microcomputer CCPU of the camera main body CMR is 
explained with reference to Figs. 1 and 6. 

When the main switch (power switch) SWMN of the 
camera main body CMR is turned on in step S101, 
electric power is supplied to the microcomputer CCPU, 
then the process proceeds to step S102 where operation 
of the camera starts. 

In step S102, the state of the switch SW1, which 
is turned on in response to the half stroke of the 
release button, is detected. If the SW1 is off, then 
the process proceeds to step S103, where an instruction 
to stop image stabilization (IS) operation (IS stop 
instruction) is transmitted to the lens LNS. 

The steps S102 and S103 are repeatedly performed 
until the switch SW1 is turned on or the main switch 
SWMN is turned off. 

When the switch SW1 is turned on in step S102, the 
process proceeds to step Sill. In step Sill, the 
microcomputer CCPU transmits an instruction to start IS 
operation (IS start instruction) to the microcomputer 
LCPU via the signal line DCL. 

Next in step S112, communication for obtaining 
parameters which are specific to the lens, such as F 
number and focal length of the lens, from the 
microcomputer LCPU is performed. 
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Then, in step S113, the luminance of the object is 
sensed by the sensor SNS, and the charging period of 
the image sensing device for obtaining image signals 
and the value for controlling the iris diaphragm are 
5 calculated in accordance with the predetermined 

exposure control program, and the microcomputer CCPU 
transmits the calculation results to the microcomputer 
LCPU. 

The process proceeds to step S114 where the focus 
10 state is detected by the sensor SNS, and an instruction 

for operating the focus lens is transmitted to the 

microcomputer LCPU. 

In step S115, the state of the pixel shifting mode 

selection switch SWSF is detected. Further, pixel 
15 shifting conditions, such as, whether or not the pixel 

shifting is to be performed and the number of image 

formation positions N SF on the image sensing device IMS 

in an image sensing operation (if it determined not to 

perform pixel shifting, N SF is set to 1, whereas if it 
20 is determined to perform pixel shifting, then N SF is 

set to at least 2), are set on the basis of the result 

of the photometry. 

Thereafter, the process proceeds to step S116, 

where the state of the switch SW2 which is turned on in 
25 response to the full stroke of the release button is 

detected. If the switch SW2 is OFF, then the process 
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returns to step Sill, and steps Sill to S115 are 
repeated. Whereas, if it is detected that the switch 
SW2 is ON, then the process proceeds to step S117. 

In step S117, a counter CNT for counting the 
5 number of image formation positions is initialized to 0. 

Then, in step S118, a timing pulse which is a 
trigger signal for image sensing operation is generated, 
and transmitted to the microcomputer LCPU. 

In step S119, the microcomputer CCPU controls the 
10 image sensing device IMS, via the driver IMDR, to 

charge, then transfer the charges in the image sensing 
device IMS . 

In step S120, the image signals read at step S119 
are temporarily stored in the RAM of the microcomputer 
15 CCPU . 

In step S121, the counter CNT 1 is increased by 1. 

In step S122, whether or not the counter CNT 
reaches the number of image formation positions N^ is 
determined. If it is not, then the process returns to 
20 step S118 and waits for the next timing pulse being 

generated, then pixel shifting operation is continued. 
If it is determined that the counter CNT has reached 
the number of image formation positions N SF , then the 
process proceeds to step S123. 
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In step S123, completion of pixel shifting 
operation (or completion of the storing of required 
image signals) is informed to the microcomputer LCPU. 

In step S124, if image signals of a plurality of 
images are stored in the RAM, then they are synthesized 
to generate a single high-resolution image, then 
outputted. Whereas, if image signals of a single image 
are stored in the RAM, it is not possible to perform 
synthesis, therefore, the image signals are outputted. 

In step S125, the image outputted in step S124 is 
stored in the memory MEM. 

Accordingly, an image sensing operation is 
completed and the process returns to step S102. If the 
switch SW1 becomes ON in step S102, then the processes 
in step Sill and the subsequent steps are repeated, 
whereas, if the switch SW1 is OFF, then an instruction 
to stop IS operation is transmitted to the 
microcomputer LCPU in step S103. 

Fig. 7 is the flowchart of control processing by 
the microcomputer LCPU of the lens LNS. 

Referring to Fig. 7, when electric power is 
provided to the exchange lens, in step S131, in 
response to the ON operation of the main switch SWMN of 
the camera main body CMR, then the process proceeds to 
step S132. 
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In step S132, whether or not the IS start 
instruction is received or not is determined, and if 
the IS start instruction is not received from the 
camera main body CMR, then the process proceeds to step 
S133. 

In step S133, whether or not the IS stop 
instruction is received from the camera main body CMR 
is determined, and if not, the process returns to step 
S132. If it is determined that the IS stop operation 
is received, then the process proceeds to step S134 
where the IS operation is stopped. More specifically, 
the pitch and yaw actuators IACTP and IACTY are 
deactivated. 

If the IS start instruction is received from the 
microcomputer CCPU while performing processes in steps 
S132 to S134, then the process proceeds from step S132 
to step S141. 

In step S141, the vibration sensors GRP and GRY 
are activated, and vibration signals in the pitch and 
yaw directions are inputted. 

Step S142 corresponds to step S112 in Fig. 6, and 
in response to requests from the microcomputer CCPU, 
the microcomputer LCPU of the lens LNS transmits the 
parameters which are specific to the lens LNS to the 
camera main body CMR. 
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In step S143, the zoom encoder ZENC and the focus 
encoder FENC are checked in order to detect zooming and 
focus states of the lens system. 

In step S144, on the basis of the detection result 
in step S143, the first coefficient for pixel shifting 
and the second coefficient for IS operation are read 
from a table stored in the ROM of the microcomputer 
LCPU. 

In step S145, the pitch and yaw actuators IACTP 
and IACTY are operated on the basis of the vibration 
signal obtained in step S141 and the second coefficient 
obtained in step S144 to reduce blurring of an image 
due to vibration. 

In step S146, the microcomputer LCPU operates the 
iris diaphragm DFM via the iris diaphragm actuator DACT 
on the basis of the information on a photometry result 
transmitted from the microcomputer CCPU to control the 
luminous exposure . 

In step S147, the focusing actuator FACT is 
operated on the basis of the information on the focus 
state detection obtained from the microcomputer CCPU to 
adjust focus. 

Next, in step S148, whether or not a timing pulse 
for triggering the pixel shifting operation is received 
is determined. If no timing pulse is received, the 
process returns to step S141, and the IS operation, the 
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Iris diaphragm control, and the focus adjustment are 
repeatedly performed. When it is determined in step 
S148 that the timing pulse is received, the process 
proceeds to step S149. 

In step S149, signals having reference waveforms 
for driving the second lens group L2 in the pitch and 
yaw directions for pixel shifting operation are 
generated by the pixel shifting signal generator 21. 
Note, the amplitudes of the signals correspond to the 
distances X p and X Y , shown in Fig. 4, in the pitch and 
yaw directions, when it is assumed that there is no 
effect of the decentering sensitivity. 

In step S150, the amplitudes of the signals having 
reference waveform, generated in step S149, are 
multiplied by the first coefficient read in step S144, 
thereby pixel shifting signals which compensates the 
effect of the decentering sensitivity of the second 
lens group L2 are generated. Thereafter, the generated 
pixel shifting signals are synthesized with signals for 
the IS operation (vibration compensation signals), 
generated by the second coefficient generator 33, in 
the synthesis circuit 41. By operating the pitch and 
yaw actuators IACTP and IACTY in accordance with the 
synthesized signals, the IS operation and the pixel 
shifting operation are performed simultaneously and 
precisely. 
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In step S151, whether or not a signal indicating 
completion of the pixel shifting operation is received 
from the microcomputer CCPU is determined, and if it is 
not, the process returns to step S148 because the pixel 
shifting has not been finished. Then, the process 
waits the next timing pulse. The processes in steps 
S148 to S150 are repeated for a predetermined number of 
times, and when the signal indicating completion of the 
pixel shifting operation is transmitted, the process 
returns from step S151 to step S132. 

Then, if the IS start instruction is not received 
in step S132 and the IS stop instruction is detected in 
step S133, then the IS operation is stopped in step 
S134; more specifically, the pitch and yaw actuators 
IACTP and IACTY are deactivated, and a series of lens 
control operation relating to the image sensing 
operation is completed. 

Fig. 8 is a timing chart for explaining operations 
of the camera main body CMR and the lens LNS shown in 
the flowcharts in Figs. 6 and 7. 

Signals A and B show states of the switches SW1 
and SW2, respectively; a signal C is a timing signal 
for pixel shifting operation; a signal D is for 
charging in the image sensing device IMS; signals E and 
F have reference waveforms for pixel shifting operation 
(referred to as "pixel shifting reference signals" 
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hereinafter) in the pitch direction and the yaw 
direction, respectively; and signals G and H are 
vibration signals, in the pitch and yaw directions, 
detected by the vibration sensors GRP and GRY, 
respectively. Here, vibration shift waveforms obtained 
by processing the detected signals by integration, for 
example, are shown. Further, signals I and J are 
operation signals for shifting the second lens group L2 
in the pitch and yaw directions, respectively. 

Next, overall operation shown in the flowcharts in 
Figs. 6 and 7 is explained with reference to the timing 
chart shown in Fig. 8. When the switch SW1 is turned 
on at time t 1# the vibration signals G and H are 
outputted. In turn, the second lens group L2 is 
operated as shown in the waveforms of the signals I and 
J in accordance with the vibration signals multiplied 
by the second coefficient. 

Then the switch SW2 is turned on at time t 2 , and 
after a predetermined period of time elapses from the 
time t 2 , a timing pulse TP1 is generated at time t lx . 
In turn, the photo- sensing elements of the image 
sensing device IMS are charged between time t 12 and 
time t 13 in accordance with a charging period 
calculated on the basis of the result of the photometry. 

When the charging operation ends at the time t 13 , 
the charges are transferred and read. At the same time, 
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the pixel shifting reference signal F in the yaw 
direction is generated. Accordingly, the second lens 
group L2 is operated in the yaw direction on the basis 
of instruction values shown by the signal J, obtained 
by adding the pixel shifting reference signal F 
multiplied by the first coefficient and the vibration 
signal H multiplied by the second coefficient. 

After a predetermined period of time has passed 
since time t 11# the second timing pulse TP2 is 
generated at time t 21 . Then, similarly to above, the 
photo-sensing elements of the image sensing device IMS 
are charged between time t 22 and time t 23 . When the 
charging operation ends at time t 23 , the charges are 
transferred and read. At the same time, the pixel 
shifting reference signal E in the pitch direction is 
generated. Accordingly, the second lens group L2 is 
operated in the pitch direction on the basis of 
instruction values shown by the signal I, obtained by 
adding the pixel shifting reference signal E multiplied 
by the first coefficient and the vibration signal G 
multiplied by the second coefficient. 

After a predetermined period of time has passed 
since time t 21 , the third timing pulse TP3 is generated 
at time t 31 . Then, similarly to above, the photo- 
sensing elements of the image sensing device IMS are 
charged between time t 32 and time t 33 . When the charging 
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operation ends at time t 33 , the charges are transferred 
and read. At the same time, the value of the pixel 
shifting reference signal F in the yaw direction is 
changed to the initial value. Accordingly, the second 
lens group L2 is operated in the yaw direction on the 
basis of instruction values shown by the signal J, 
corresponding to the vibration signal H multiplied by 
the second coefficient. 

After a predetermined period of time has passed 
since time t 31 , the last timing pulse TP4 is generated 
at time t 41 . Then, similarly to above, the photo- 
sensing elements of the image sensing device IMS are 
charged between time t 42 and time t 43 . When the charging 
operation ends at time t 43 , the charges are transferred 
and read. At the same time, the value of the pixel 
shifting reference signal E in the pitch direction is 
changed to the initial value. Accordingly, the second 
lens group L2 is operated in the pitch direction on the 
basis of instruction values shown by the signal I, 
corresponding to the vibration signal G multiplied by 
the second coefficient. 

After the switch SW1 is turned off at time t s , the 
vibration detection and the operation of the second 
lens group L2 are stopped. 

The given point of an image on the image sensing 
device IMS while performing the image stabilization and 
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the pixel shifting operation at time t 11# t 21 , t 31 , t 41 
and t 5 are at IM1, IM2, IM3, IM4 and IM1 in Fig. 4, 
respectively, thus image formation positions of the 
image are shifted by a half pixel distance from each 
other in the vertical and horizontal directions. 

Note, the reason for the pixel shifting reference 
waveform being a trapezoid shape rather than a square 
shape is to mitigate shock of sudden movement of the 
second lens group L2 . 

Next, the principle for generating an image signal 
of a single high-resolution image by synthesizing a 
plurality of images obtained while performing pixel 
shifting operation is explained with reference to Figs. 
9A, 9B and 10A to 10E. 

Figs. 9A and 9B are views for explaining relative 
position relationship between an image and the image 
sensing device IMS in pixel shifting operation. Fig. 
9A corresponds to Fig. 4, and it shows that the 
position of the image shifts in the order of IM1, IM2, 
IM3, IM4, and IM1 with respect to the pixels of the 
image sensing device IMS fixed in the camera main body 
CMR. The above movement is equivalent to a case where 
the position of the image sensing device IMS moves in 
the order of IG1, IG2 , IG3, IG4, and IG1 with respect 
to a fixed object. Now, let an output signal from each 
pixel when the image sensing device IMS is at the 



-48- 



position, IG1 # be IGl(i, j), where i and 3 are 
coordinates of the pixel. The image sensing device IMS 
is an area sensor having m x n pixels. Similarly, let 
output signals when image sensing device IMS is at the 
positions, IG2, IG3 and IG4 , be IG2(i, j ) , IG3(i, 3) , 
and IG4(i, 3), respectively. 

Figs. 10A to 10E show how to synthesize these four 
sets of image signals. 

Let a new set of image signals representing 2m x 
2n pixels obtained by four sets of m x n pixels be 
denoted by IMG(u, v) . The four left uppermost corner 
pixels of the image signals IMG(u, v) are obtained by 
synthesizing the respective left uppermost corner 
pixels of the four original images as shown in Figs. 
10A to 10E. When a method for synthesizing images is 
considered with reference to Figs. 10A to 10E, it is 
possible to obtain an image signal of a single high- 
resolution image from the four original images by using 
the following four equations; 



IMG(u = 21-1, v = 2j) 
IMG(u = 2i-l, v = 2j-l) 
IMG(u = 2i, v = 2j-l) 
IMG(u = 2i, v = 23) 



IGl(i, 3) ...(7) 

<- IG2(i, 3) ...(8) 

IG3(i, 3) ...(9) 

— IG4(i, 3) ...(10) 
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Note, the aforesaid pixel shifting operation and 
the image synthesis method is used when image signals 
are obtained from a black-and-white image sensing 
device and a multiple-CCD type color image sensing 
device using a color separation prism. When image 
signals are obtained from a single CCD type color image 
sensing device covered with a mosaic color filter, 
although there are little differences in pixel shifting 
amount in the pixel shifting operation and the image 
synthesis method, the basic ideas of the pixel shifting 
operation and the image synthesis are the same. 

According to the first embodiment as described 
above , 

(1) By changing a pixel shifting signal in 
accordance with the first coefficient, changing a 
vibration signal in accordance with the second 
coefficient, and operating an image stabilization lens 
system on the basis of the synthesized signal of the 
above two changed signals, it is possible to perform 
image stabilization operation and pixel shifting 
operation at the same time using only a single image 
shifting means, i.e., the image stabilization lens 
system. Accordingly, it is possible to obtain a high- 
resolution image, by pixel shifting operation, with 
less deterioration due to vibration. 
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(2) By using the first and second coefficients 
selected in accordance with zooming and focus states, 
it is possible to always perform precise image 
stabilization operation and pixel shifting control even 

5 when the zooming and focus states are changed. 

(3) Since synthesis of images obtained while 
performing pixel shifting operation is performed within 
a camera, an image signal of a high-resolution image 
can be obtained without using an exclusive external 

10 device . 

< Second Embodiment > 

The first embodiment is for precisely and 
simultaneously performing image stabilization and pixel 

15 shifting operation. In the following second embodiment, 
an optimum pixel shifting mode is selected in 
accordance with the state of the camera. 

Fig. 11 is a flowchart showing control processing 
in the camera main body CMR according to the second 

20 embodiment; Figs. 12A and 12B are views for explaining 
relative position relationship between an image and the 
image sensing device IMS in pixel shifting operation 
according to the second embodiment; and Figs. 13A to 
13C are views for explaining a principle of image 

25 synthesis in a second pixel shifting mode (will be 
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explained later). The second embodiment will be 
explained with reference to accompanying drawings. 

The control processing in the camera main body CMR 
in the second embodiment is basically the same as that 
5 shown in Fig. 6 explained in the first embodiment. 

However, the process performed in step S115, i.e., "to 
set a pixel shifting condition" is realized by a sub- 
routine as shown in Fig. 11, thereby the advantage of 
O the second embodiment can be obtained. Below, the 

Lfl 10 control processing according to the second embodiment 

yg will be explained with reference to Figs. 6 and 11. 

m Since the processes shown in Fig. 6 have been already 

q explained in detail in the first embodiment, they are 

j^i only briefly explained in the second embodiment. 

15 Referring to Fig. 6, when it is determined in step 

w " S102 that the switch SW1 is ON, then the process 

proceeds to step Sill. Thereafter, an IS start 
instruction is transmitted to the lens LNS in step Sill 
and parameters are received from the lens LNS in step 
20 S112. Next in steps S113 and S114, photometry and 

focus state detection are performed, and the obtained 
results are transmitted to the microcomputer LCPU of 
the lens LNS. 

In the next step, S115, the processes shown in Fig. 
25 11 are performed. In step S215 in Fig. 11, the 

microcomputer CCPU requests transmission of the peak 
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value of vibration angular velocity, co peaX# in a 
predetermined period of time to the microcomputer LCPU. 
In turn, the microcomputer LCPU transmits the peak 
value of the vibration angular velocity o> peak occurring 
5 in a two second interval, for example, to the 
microcomputer CCPU. 

In step S216, the maximum vibration value 8 while 
exposing the image sensing device IMS when the IS 
function is not operated is calculated on the basis of 
10 the following equation, 

8 = f x co peak x t exp . . . (11) 

where, f denotes a focal length of the optical system, 
15 and t exp is an exposure time, i.e., charging period, of 
III the image sensing device determined on the basis of the 

result of photometry. The maximum vibration value 8 
obtained here is used in the subsequent steps as an 
index of vibration for determining whether or not pixel 
20 shifting operation should be performed. 

In step S217, the value of the maximum vibration 
value 8 is checked. If the maximum vibration value 8 
is equal or less than a predetermined value DELI , then 
it is determined that effect of vibration is small, 
25 thus the quality of an image would improve by 

performing pixel shifting operation. Accordingly, the 
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process proceeds to step S218 and the number of image 
formation positions N SF , is set to four. Here, the 
number of image formation positions, four, indicates 
the same pixel shifting operation and image synthesis 
5 explained in the first embodiment. This overall 

operation is referred to as "first pixel shifting mode" 
in the second embodiment. 

Whereas, if it is determined in step S217 that the 
vibration value 6 is larger than the value DELI and 

10 equal or less than a predetermined value DEL 2 (DEL 2 > 
DELI), then the process proceeds to step S219 where the 
number of image formation positions N SF is set to two. 
This is because deterioration of an image is expected 
to some degree even if the image stabilization is 

15 performed. Therefore, a mode which requires less image 
formation positions is selected (this mode is referred 
to as "second pixel shifting mode") so as to reduce 
deterioration of the image quality due to vibration and 
to realize an optimum improvement in image quality. 

20 Details of the second pixel shifting mode are explained 
later. 

In step S217, when it is determined that the 
vibration value 5 is greater than the predetermined 
value DEL2 , then in step S220, the number of image 
25 formation positions N SF is set to 1 indicating that no 
pixel shifting operation is to be performed. The 
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reason for setting the number of image formation 
positions N SF to 1 is that the effect of the vibration 
which causes deterioration of an image is stronger than 
the effect of pixel shifting operation which improves 
5 the quality of the image. Therefore, the pixel 
shifting is disabled. 

After one of the processes in steps S218 to S220 
is performed, the process proceeds to step S221. 

In step S221, information on the determined pixel 
10 shifting mode is transmitted to the microcomputer LCPU 
of the lens LNS • 

In step S222, the type of the pixel shifting mode 
is displayed on the display device DISP of the camera 
main body CMR to inform the user of which mode is used 
15 for photographing an image. 

After step S222 is completed, the process returns 
to step S116. 

In step S116, determination of the state of the 
switch SW2 is performed, and if SW2 is ON, then the 
20 process proceeds to step S117. 

In steps S117 to S122, the pixel shifting is 
performed as explained in the first embodiment, however, 
in the second embodiment, the pixel shifting is 
performed in accordance with the selected pixel 
25 shifting mode determined in steps S215 to S222. 

Therefore, the pixel shifting operation is performed in 
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the selected pixel shifting mode if either the first or 
second pixel shifting mode is selected. Whereas, if it 
is determined in the steps S215 to S222 not to perform 
pixel shifting operation, image formation position is 
not shifted. 

Then, in step S123, the completion of the pixel 
shifting operation (or completion of the storing of 
required image signals) is informed to the 
microcomputer LCPU, and the process proceeds to step 
S124. In step S124, the image synthesis processing 
corresponding to the selected pixel shifting mode is 
performed if the first or second pixel shifting mode is 
selected. If the pixel shifting operation was not 
performed, there is no need to perform image synthesis, 
therefore, the photographed image is directly outputted. 

Then, in step S125, the obtained image is recorded 
and the process returns to step S102. 

Figs. 12A and 12B are views for explaining pixel 
shifting method when the number of image formation 
positions is two. Fig. 12A shows shift of an image 
with respect to the image sensing device IMS. As shown 
in Fig. 12A, the image sensing device IMS is charged 
and read when a given point of an image is at a 
position IM21. Then, after the point of the image is 
shifted to the position IM22, the image sensing device 
is charged and read again. Thereafter, the image 
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formation position is moved back to the initial 
position so that the point of the image is formed at 
the position IM21. 

Fig. 12B shows a movement of the image sensing 
device IMS equivalent to the aforesaid movement of the 
image shown in Fig, 12A, and a pixel of the image 
sensing device IMS which is at an initial position IG21 
is moved to the position IG22 by pixel shifting 
operation, then moved back to the initial position IG21. 

Fig, 13 is a view for explaining an image 
synthesis method in the second pixel shifting mode. In 
the second pixel shifting mode, on the basis of image 
signals of two images, IG21(i, j ) and IG22(i, j), 
namely, image signals of (2 x m x n) pixels, an image 
IMG2(u, v) expressed with (4 x m x n) pixels is 
obtained. For this reason, pixels expressed by black 
dots in IMG2(u, v) are the same values as pixels in the 
images IG21(i, j) and IG22(i, j ) , and the pixels shown 
by white dots are interpolated with averages of the 
values of the neighboring four pixels (at edge, two or 
three pixels, instead of four pixels). 

In equations, 

IMG2(u=2i-l, v=2j) IG21(i, j) ...(12) 

IMG2(u=2i, v=2j-l) IG22(i, j) ...(13) 
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IMG2(u=2i-l, v=2j-l) «- {IG21(i, 3) + IG22(i # 

3) + IG21(i, 3-1) + IG22(i-l, j)}/4 ...(14) 

IMG2(u=2i, v=2j) {IG21(i, 3) + IG22(i, j) + 

IG21(i+l, j) + IG22(i, j + l)}/4 .,.(15) 

According to the second embodiment as described 
above, in addition to the same effects as those of the 
first embodiment, 

(4) It is possible to perform the optimum pixel 
shifting operation in accordance with image sensing 
conditions since the optimum image sensing mode is 
selected in accordance with the vibration in 
consideration of possibility of deteriorating image 
quality due to vibration while performing the pixel 
shifting operation and effect of the pixel shifting 
operation on resolution of the image. 

< Third Embodiment > 

In the second embodiment, the optimum pixel 
shifting mode is selected in accordance with vibration. 
In the third embodiment, whether or not the pixel 
shifting operation is to be performed is determined in 
accordance with an image sensing mode set by a user, 
and, if it is determined to perform pixel shifting 
operation, the pixel shifting mode is changed in 
accordance with the image sensing mode. 
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The control processing in the camera main body CMR 
in the third embodiment is basically the same as that 
shown in Fig. 6 explained in the first embodiment, 
similarly to the second embodiment. However, the 
process performed in step S115, i.e., "to set a pixel 
shifting condition" is realized by a sub-routine as 
shown in Fig. 14, thereby the advantage of the third 
embodiment can be obtained. Below, the control 
processing according to the third embodiment will be 
explained with reference to Figs. 6 and 14. Since the 
processes shown in Fig. 6 have been already explained 
in detail in the first embodiment, they are only 
briefly explained in the second embodiment. 

Referring to Fig. 6, when it is determined in step 
S102 that the switch SW1 is ON, the process proceeds to 
step Sill. Thereafter, an IS start instruction is 
transmitted to the lens LNS in step Sill and parameters 
are received from the lens LNS in step S112. Next in 
steps S113 and S114, photometry and focus state 
detection are performed, and the obtained results are 
transmitted to the microcomputer LCPU of the lens LNS. 

In the next step, S115, the processes shown in Fig. 
14 are performed. In step S3 15 in Fig. 14, the status 
of the image mode selection switch SWMOD (image serising 
mode) provided in the camera main body CMR is 
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determined, thereby image sensing conditions, such as 
exposure control mode, set by the user are determined . 

In step S316, whether or not the image sensing 
mode selected by the user is a landscape mode is 
determined. Landscape mode is an exposure control mode 
in which field depth is deepened by setting a small 
iris diaphragm control value (large F number). When 
the landscape mode is set, it is expected that the 
object stands still, and the camera is held still, thus 
vibration would not occur in most cases. Accordingly, 
the process proceeds to step S319, where a "high- 
resolution" mode for sensing four images while shifting 
between four image formation positions in pixel 
shifting operation is set. When it is determined in 
step S316 that the set image sensing mode is not the 
landscape mode, then the process proceeds to step S317. 

In step S317, whether or not the selected image 
sensing mode is a portrait mode is determined. 
Portrait mode is an exposure control mode in which 
field depth is narrowed by setting the iris diaphragm 
control value to near open (small F number). Since it 
is expected that the conditions for photographing in 
the portrait mode is similar to those of the landscape 
mode, the process proceeds to step S319. Whereas, if 
it is determined in step S317 that the set image 
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sensing mode is not the portrait mode, then the process 
proceeds to step S318. 

In step S318, whether or not the set image sensing 
mode is a sport mode is determined. Sport mode is an 
exposure control mode for photographing a moving object 
as if it is not moving by shortening exposure time. 
When the sport mode is selected, it is expected that 
the object is moving and the camera may be panning. In 
other words, the movement of the camera similarly to 
the vibration by a large displacement amount is 
expected. Further, blurring of the object due to the 
movement of the object while performing pixel shifting 
operation is expected. Accordingly, improvement of 
image quality is not anticipated; on the contrary, the 
obtained image by performing pixel shifting operation 
would be unnatural. Therefore, when the sport mode is 
set, the process proceeds to step S321 where N SF is set 
to one and the pixel shifting is disabled. If it is 
determined in step S318 that the sport mode is not set, 
namely, when the set image sensing mode is not any of 
the landscape, portrait, and sport modes, the process 
proceeds to step S320, and the number of image 
formation positions N SF is set to two. 

After one of the processes in steps S319 to S321 
is performed, the process proceeds to step S322. 
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In step S322, information on the determined pixel 
shifting mode is transmitted to the microcomputer LCPU 
of the lens LNS . 

In step S323, the type of the pixel shifting mode 
5 is displayed on the display device DISP of the camera 
main body CMR to inform the user of which mode is used 
for photographing an image. 

After step S323 is completed, the process returns 
to step S116. 

10 In step S116, determination of the state of the 

switch SW2 is performed, and if SW2 is ON, then the 
process proceeds to step S117. 

In steps S117 to S122, the pixel shifting is 
performed in accordance with the selected pixel 

15 shifting mode as explained in the second embodiment. 

Then, in step S123, the completion of the pixel 
shifting operation (or completion of the storing of 
required image signals) is informed to the 
microcomputer LCPU, and the process proceeds to step 

20 S124. In step S124, the image synthesis processing 
corresponding to the selected pixel shifting mode is 
performed as described in the second embodiment . 

In step S125, the obtained image is recorded and 
the process returns to step S102. 
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According to the third embodiment as described 
above, in addition to the same effects as those of the 
first embodiment, 

(5) It is possible to perform an optimum pixel 
shifting operation suitable for movements of both a 
camera and an object by determining whether or not it 
is appropriate to perform pixel shifting operation and 
changing pixel shifting modes, in accordance with an 
image sensing mode set by the user. 

Further, an image sensing mode is selected on the 
basis of different photographing situations which 
require different exposure control, however, an image 
sensing mode may be selected on the basis of the result 
of focus state detection. 

<Fourth Embodiment > 

In the first embodiment, the pixel shifting 
operation and the image stabilization are achieved 
using one shifting device, namely, the second lens 
group L2, which is operated in accordance with the 
shift amounts adjusted by the first coefficient for the 
pixel shifting operation and the second coefficient for 
the image stabilization. It is possible to use the 
first coefficient in an image stabilization method 
which is different from the aforesaid image 
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stabilization method in order to further improve image 
stabilization ability. 

Fig, 15 is a block diagram illustrating a detailed 
configuration of microcomputers CCPU and LCPU of the 
camera main body CMR and the lens LNS and their 
peripheral units according to the fourth embodiment of 
the present invention. This is a modified version of 
the block diagram shown in Fig. 5. As shown in Fig. 15, 
a movement vector detector 17 is added to the 
microcomputer CCPU of the camera main body CMR and a 
third coefficient generator 22a is added to the 
microcomputer LCPU of the lens LNS. 

The movement vector detector 17 is a known circuit 
for detecting shifted amount, due to vibration, between 
two images sensed at different times, on the basis of a 
spatial correlation of image signals of the two images, 
and used in, so-called, electronic image stabilization. 
The two images used for movement vector detection are 
those obtained while performing pixel shifting 
operation or those obtained periodically with no 
relation to pixel shifting operation. Alternately, a 
signal from the focus state detection sensor may be 
used. Note, image formation positions of the two 
images obtained in pixel shifting operation are 
naturally shifted by a predetermined amount, therefore, 
it is necessary to correct the shifted amount in 
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consideration of the shift amount due to the pixel 
shifting operation for detecting a movement vector. 

Since an image stabilization (IS) system including 
vibration detection sensors, such as vibration -type 
gyroscopes, utilizing inertia is provided to the camera 
in the fourth embodiment, while the IS system is 
operated, theoretically, blurring of an image is 
corrected, and thus the movement vector detector 17 
does not detect vibration. However, the vibration 
detection sensors, such as the vibration- type gyro, 
utilizing inertia have a defect that it can not detect 
vibration of a very low frequency range because of 
direct -current offset and drift of an output signal, 
for instance. Therefore, the movement vector detector 
17 detects low frequency vibration while the IS system 
is operated. Thus, the vibration signal detected by 
the movement vector detector 17 is converted by the 
third coefficient generator 22a, and the converted 
value, the shift amount instruction value, calculated 
by the first coefficient generator 22, for the second 
lens group L2 for pixel shifting operation, and the 
shift amount instruction value, calculated by the 
second coefficient generator 33, for the second lens 
group L2 for image stabilization are synthesized in the 
synthesis circuit 41. Then, by operating the IS 
actuator 42 on the basis of the synthesized signals , 
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the IS system capable of compensating vibration in a 
wide frequency range, from a low frequency to a high 
frequency is realized, thereby image stabilization 
ability is improved. Accordingly, blurring of images 
obtained while performing pixel shifting operation is 
reduced, thereby contributing to improvement of 
resolution of an image. 

Figs. 16 and 17 are flowcharts of control 
processings by the microcomputers in the camera main 
body CMR and in the lens LNS, respectively, according 
to the fourth embodiment of the present invention. 

First, a flow of the control processing by the 
microcomputer CCPU of the camera main body CMR is 
explained with reference to Figs. 1 and 16. 

When the main switch (power switch) SWMN of the 
camera main body CMR is turned on in step S401, 
electric power is supplied to the microcomputer CCPU, 
then the process proceeds to step S402 where operation 
of the camera starts. 

In step S402, the state of the switch SW1, which 
is turned on in response to the half stroke of the 
release button, is detected. If the SW1 is off, then 
the process proceeds to step S403 where an instruction 
to stop image stabilization (IS) operation (IS stop 
instruction) is transmitted to the lens LNS. 
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The steps S402 and S403 are repeatedly performed 
until the switch SW1 is turned on or the main switch 
SWMN is turned off. 

When the switch SW1 is turned on in step S402, the 
process proceeds to step S411. In step S411, the 
microcomputer CCPU transmits an instruction to start IS 
operation (IS start instruction) to the microcomputer 
LCPU via the signal line DCL . 

Next in step S412, communication for obtaining 
parameters which are specific to the lens, such as F 
number and focal length of the lens, from the 
microcomputer LCPU is performed. 

Then, in step S413, the luminance of the object is 
measured by the sensor SNS, and the charging period for 
obtaining image signals from the image sensing device 
and the iris diaphragm control value are calculated, 
and the microcomputer CCPU transmits the calculation 
result to the microcomputer LCPU. 

In step S414, the focus state is detected by the 
sensor SNS, and the result is also transmitted to the 
microcomputer LCPU. 

In step S415, the state of the pixel shifting mode 
selection switch SWSF is detected as well as pixel 
shifting conditions, e.g., whether or not to perform 
pixel shifting operation and the number of image 
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formation positions, are set on the basis of the result 
of the photometry performed in step S413, for example. 

In step S416, the microcomputer CCPU controls the 
image sensing device IMS # via the driver IMDR, to 
charge, then transfer the charges in the image sensing 
device IMS. 

In step S417, the image signals read at step S416 
are temporarily stored in the RAM of the microcomputer 
CCPU. 

In step S418, a movement vector is determined from 
image signals of two images stored in the RAM. Note, 
when the process in step S418 is performed for the 
first time, image signals representing only one image 
are stored in the RAM; therefore, "0" is outputted as 
the movement vector. 

In step S419, the movement vector determined in 
step 418 is transmitted to the microcomputer LCPU. 

Thereafter, the process proceeds to step S420, 
where the state of the switch SW2 which is turned on in 
response to the full stroke of the release button is 
detected. If the switch SW2 is OFF, then the process 
returns to step S411, and steps S411 to S419 are 
repeated. Whereas, if it is detected that the switch 
SW2 is ON, then the process proceeds to step S421. 
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In step S421, the same pixel shifting control as 
that performed in steps S117 to S122 in Fig. 6 
explained in the first embodiment is performed. 

In step S422, completion of pixel shifting 
operation (or completion of the storing of required 
image signals) is informed to the microcomputer LCPU. 

In step S423, if image signals of a plurality of 
images are stored in the RAM, then they are synthesized 
to generate a single high-resolution image, then 
outputted. Whereas, if image signals of a single image 
are stored in the RAM, it is not possible to perform 
synthesis, therefore, the image signals are outputted. 

In step S424, the image outputted in step S423 is 
stored in the memory MEM. 

Accordingly, an image sensing operation is 
completed and the process returns to step S402. If the 
switch SW1 becomes ON in step S402, then the processes 
in step S411 and the subsequent steps are repeated, 
whereas, if the switch SW1 is OFF, then an instruction 
to stop IS operation is transmitted to the 
microcomputer LCPU in step S403. 

Fig. 17 is the flowchart of control processing by 
the microcomputer LCPU of the lens LNS. 

Referring to Fig. 17, when electric power is 
provided to the exchange lens r in step S431, in 
response to the ON operation of the main switch SWMN of 
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the camera main body CMR, then the process proceeds to 
step S432. 

In step S432, whether or not the IS start 
instruction is received or not is determined, and if 
the IS start instruction is not received from the 
camera main body CMR, then the process proceeds to step 
S433. 

In step S433, whether or not the IS stop 
instruction is received from the camera main body CMR 
is determined, and if not, the process returns to step 
S432. If it is determined that the IS stop operation 
is received, then the process proceeds to step S434 
where the IS operation is stopped. More specifically, 
the pitch and yaw actuators IACTP and IACTY are 
deactivated. 

If the IS start instruction is received from the 
microcomputer CCPU while performing processes in steps 
S432 to S434, then the process proceeds from step S432 
to step S441. 

In step S441, the vibration sensors GRP and GRY 
are activated, and vibration signals in the pitch and 
yaw directions are inputted. 

Step S142 corresponds to step S412 in Fig. 16, and 
in response to requests from the microcomputer CCPU, 
the microcomputer LCPU of the lens LNS transmits the 
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parameters which are specific to the lens LNS to the 
camera main body CMR. 

In step S443, the zoom encoder ZENC and the focus 
encoder FENC are checked in order to detect zooming and 
focus states of the lens system. 

In step S444, on the basis of the detection result 
in step S443, the first coefficient for pixel shifting 
and the second coefficient for IS operation are read 
from a table stored in the ROM of the microcomputer 
LCPU. 

In step S445, the pitch and yaw actuators IACTP 
and IACTY are operated on the basis of the vibration 
signals obtained in step S441 and the second 
coefficient obtained in step S444 to reduce blurring of 
an image due to vibration. 

In step S446, the microcomputer LCPU operates the 
iris diaphragm DFM via the iris diaphragm actuator DACT 
on the basis of the information on a photometry result 
transmitted from the microcomputer CCPU to control the 
luminous exposure . 

In step S447, the focusing actuator FACT is 
operated on the basis of the information on the focus 
state detection obtained from the microcomputer CCPU to 
adjust focus. 

In step S448, the movement vector transmitted in 
step S419 in Fig. 16 is received. 
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In step S449, shift amount instruction values for 
the second lens group L2 for compensating blurring are 
generated on the basis of the movement vector signal 
obtained in step S448 and the first coefficient 
obtained in step S444, and further added to shift 
amount instruction values obtained from the vibration 
sensors GRP and GRY and the second coefficient obtained 
in step S444. Then, the pitch and yaw actuators IACTP 
and IACTY are driven in accordance with the added 
signal, thereby realizing image stabilization in wide 
frequency range of vibration. 

Next, in step S450, whether or not a timing pulse 
for triggering the pixel shifting operation is received 
is determined. If no timing pulse is received, the 
process returns to step S441, and the IS operation, the 
iris diaphragm control, the focus adjustment, and 
reception of the moving vector are repeatedly performed. 
When it is determined in step S450 that the timing 
pulse is received, the process proceeds to step S451. 

In step S451, signals having reference waveforms 
for driving the second lens group L2 in the pitch and 
yaw directions for pixel shifting operation are 
generated by the pixel shifting signal generator 21. 

In step S452, the amplitudes of the signals having 
reference waveform, generated in step S451, are 
multiplied by the first coefficient read in step S444, 
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thereby pixel shifting signals which compensates the 
effect of the decentering sensitivity of the second 
lens group L2 are generated. Thereafter, the generated 
pixel shifting signals are synthesized with signals for 
the IS operation (vibration compensation signals), 
generated in step S449, in the synthesis circuit 41. 
By operating the pitch and yaw actuators IACTP and 
IACTY in accordance with the synthesized signals, the 
IS operation and the pixel shifting operation are 
performed simultaneously and precisely. 

In step S453, whether or not a signal indicating 
completion of the pixel shifting operation is received 
from the microcomputer CCPU is determined, and if it is 
not, the process returns to step S450 because the pixel 
shifting is not finished. Then, the process waits the 
next timing pulse. The processes in steps S450 to S453 
are repeated for a predetermined number of times, and 
when the signal indicating completion of the pixel 
shifting operation is transmitted, the process returns 
from step S453 to step S451. 

Then, if the IS start instruction is not received 
in step S432 and the IS stop instruction is detected in 
step S433, then the IS operation is stopped in step 
S434; more specifically, the pitch and yaw actuators 
IACTP and IACTY are deactivated, and a series of lens 
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control operation relating to the image sensing 
operation is completed. 

According to the fourth embodiment as described 
above, in addition to the same effects as those of the 
5 first embodiment, 

(6) Correctable frequency range of vibration is 
widened by converting a pixel shifting signal using the 
first coefficient , converting a movement vector signal 
O using the first coefficient, converting a vibration 

UT 10 signal using the second coefficient, and driving the 

*y optical system for image stabilization in accordance 

111 with a synthesized signal of the above three converted 

p signals . 

fy (7) Precise vibration compensation is realized 

p 15 even when zooming and focus states are changed by using 

the first and second coefficients corresponding to the 

zooming and focus states. 

Note, the advantage of the fourth embodiment is 

obtained with or without pixel shifting function. 

20 

<Modif ications> 

In the first to fourth embodiments, a lens group 
in the optical system is moved in the orthogonal 
direction with respect to the optical axis of the 
25 optical system, thereby used as an image shifting means 
for realizing image stabilization and pixel shifting 
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operation by utilizing optical shifting feature of the 
lens group. Alternately, it is possible to use a pair 
of transparent plates between which transparent liquid 
is filled, so-called, a variable apical angle prism is 
used. 

Further, any type of optical systems may be used 
as the optical system in the second and third 
embodiments. In addition, the features of the second 
and third embodiments are achieved regardless of the 
existence of image stabilization function. 

<Fifth Embodiment > 

Fig. 18 is a flowchart of control processing 
performed by the microcomputer CCPU of the camera main 
body according to a fifth embodiment of the present 
invention. The processes in Fig. 18 are same as those 
in Fig. 6 explained in the first embodiment except step 
S115. 

In step S1115 in Fig. 18, the state of the pixel 
shifting mode selection switch SWSF is detected, and 
image sensing conditions, such as the type of pixel 
shifting operation and the type of luminous exposure 
control of the image sensing device, are set on the 
basis of results of photometry and focus state obtained 
in steps S113 and 114. The detail will be explained 
later . 
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Note, the control processing by the microcomputer 
LCPU of the lens LNS is the same as the one shown in 
Fig. 7 explained in the first embodiment. 

Next, method for controlling luminous exposure 
according to the fifth embodiment will be explained in 
detail with reference to Figs. 19 to 21. First, 
referring to Fig. 20, a typical photo-sensing 
characteristics of an image sensing device is explained. 

Lines in a graph shown in Fig. 20 show 
characteristics of a film and an image sensing device, 
and the abscissa indicates luminous exposure and the 
ordinate indicates transparency of image recorded on 
the film or output voltage of the image sensing device. 
If an object is sensed with a fixed iris diaphragm 
control value at a fixed shutter speed, the abscissa 
can be considered as luminance of the object. 

In Fig. 20, a broken line represents the 
characteristics of a silver-halide film, and a solid 
line represents the characteristics of the image 
sensing device, such as CCD. While the silver-halide 
film has a wide dynamic range, the dynamic range of the 
image sensing device is narrow, and the image sensing 
device can only reproduce an image of the object in a 
luminance range between log HI and log H2 . 
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Fig. 19 is a flowchart showing processes for 
setting luminous exposure, which shows detailed 
processes in step S1115 in Fig. 18. 

In step S181, the image of the object whose 
luminance is measured in step S113 is divided into a 
plurality of blocks. Method for dividing the image is 
explained with reference to Fig. 21. 

Fig. 21 is a view showing an area where focus 
state detection and the photometry, performed by the 
sensor SNS shown in Fig. 1, are performed. The 
detection area of the sensor SNS is roughly the same as 
the photo- sensing area of the image sensing device IMS, 
and is divided into 96 (= 8 x 12) areas which are 
surrounded by thin lines in Fig. 21. Focus state and 
luminance can be individually detected in each divided 
area. Note, the sensor SNS can be realized by a pair 
of secondary focusing optical systems and focus state 
detection means, configured with two-dimensional image 
sensor, provided in respective secondary optical 
systems, adopting secondary phase difference detection 
method, for example. 

On the sensor SNS, an image of an object is formed, 
as shown in Fig. 21. On the basis of the results of 
focus state detection and photometry performed on the 
image, the image is divided into five blocks, BK1 to 
BK5, surrounded by bold lines in Fig. 21, in this case. 
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The division of the area is performed so that each 
block consists of the areas having roughly the same 
focus state and luminance levels out of the aforesaid 
96 areas. 

In step S182 in Fig. 19 , a block including a main 
object is inferred from the divided blocks BK1 to BK5 
on the basis of a predetermined algorithm. More 
specifically, the maim object can be inferred using the 
following principles: 

i) If an area on which focus state detection is 
to be performed is designated by a user, then an object 
in the area is the main object; 

ii) If the camera is set to a mode for 
automatically detecting a main object, then an object 
which is near the center of an image and at relatively 
short distance from the camera is the main object; 

iii) If an area on which photometry is to be 
performed is designated by the user, then an object 
included in the area is the main object; and 

iv) If a camera has a function for detecting the 
direction of the line of sight of the user, then an 
object on the line of sight is the main object. 

In the fifth embodiment, one of the above 
principles is used, and it is assumed that a person 
included in the block BK3 is determined as the main 
object . 
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Then in step S183, on the basis of the divided 
blocks and the result of the main object inference, the 
blocks are made into groups. Since the main purpose of 
the fifth embodiment is luminous exposure control while 
performing pixel shifting operation, the number of 
luminous exposures used for sensing images in the pixel 
shifting operation should be equal or less than the 
number of image formation positions. Accordingly, the 
blocks are further collected into groups of a smaller 
number; more specifically, the number which is equal or 
less than that of the image formation positions in the 
pixel shifting operation. For example, the blocks BK1 
and BK2 form a high luminance group GP1 , and the block 
BK3, which includes the main object, forms a medium 
luminance group GP2, and the groups BK4 and BK5 form a 
low luminance group GP3« Namely, 

BK1, BK2 -> GP1 (high luminance group) 
BK3 -* GP2 {medium luminance group) 

BK4, BK5 -* GP3 (low luminance group) 
Next, in step S184, an average luminance of each 
group determined in step SI 8 3 is calculated. 

In step S185, proper luminous exposures E L to E n 
(n=3, in this case) corresponding to respective average 
luminances are calculated. Then in step S186, iris 
diaphragm control values and exposure times t el to t €n 
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for obtaining the proper luminous exposures E ± to E n 
are calculated on the basis of a predetermined program. 

After step S186, the process returns to step S116 
in Fig. 18 , and the pixel shifting operation and 
exposure (charging) of the image sensing device IMS are 
performed. 

Fig. 22 is a timing chart for explaining 
operations of the camera main body CMR and the lens LNS 
shown in the flowcharts in Figs. 18, 19 and 7. 

Signal a and b show states of the switches SW1 and 
SW2, respectively; a signal c is a timing signal for 
pixel shifting operation; a signal d is for charging in 
the image sensing device IMS; signals e and f have 
reference waveforms for pixel shifting operation 
(referred to as "pixel shifting reference signals" 
hereinafter) in the pitch direction and the yaw 
direction, respectively; and signals <j and h are 
vibration signals, in the pitch and yaw directions, 
detected by the vibration sensors GRP and GRY, 
respectively. Here, vibration shift waveforms obtained 
by processing the detected signals by integration, for 
example, are shown. Further, signals i and J. are 
operation signals for shifting the second lens group L2 
in the pitch and yaw directions, respectively. 
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Next, overall operation shown in the flowcharts in 
Figs. 18, 19 and 7 is explained with reference to the 
timing chart shown in Fig. 22. 

When the switch SW1 is turned on at time t 1# the 
5 vibration signals a and h are outputted. In turn, the 
second lens group L2 is operated as shown in the 
waveforms of the signals i and J in accordance with the 
vibration signals multiplied by the second coefficient. 
Then the switch SW2 is turned on at time t 2 , and 

10 after a predetermined period of time elapses from the 
time t 2 , a timing pulse TP1 is generated at time t u . 
In turn, the photo- sensing elements of the image 
sensing device IMS are charged between time t 12 and 
time t 13 , namely, the exposure time t e2 , calculated in 

15 step S186 in Fig. 19, which is suitable for the medium 
luminance group GP2 . 

When the charging operation ends at the time t 13 , 
the charges are transferred and read. At the same time, 
the pixel shifting reference signal f in the yaw 

20 direction is generated. Accordingly, the second lens 
group L2 is operated in the yaw direction on the basis 
of instruction values shown by the signal J, obtained 
by adding the pixel shifting reference signal f 
multiplied by the first coefficient and the vibration 

25 signal h, multiplied by the second coefficient. 



-81- 



After a predetermined period of time has passed 
since time t llf the second timing pulse TP 2 is 
generated at time t 21 . Then, similarly to above, the 
photo-sensing elements of the image sensing device IMS 
are charged between time t 22 and time t 23 . The exposure 
time used this time is the exposure time t el , 
calculated in step S186 in Fig. 19, which is suitable 
for the high luminance group GP1. 

When the charging operation ends at time t 23 , the 
charges are transferred and read- At the same time, 
the pixel shifting reference signal e in the pitch 
direction is generated. Accordingly, the second lens 
group L2 is operated in the pitch direction on the 
basis of instruction values shown by the signal i 
obtained by adding the pixel shifting reference signal 
e multiplied by the first coefficient and the vibration 
signal g multiplied by the second coefficient. 

After a predetermined period of time has passed 
since time t 21 , the third timing pulse TP 3 is generated 
at time t 31 . Then, similarly to above, the photo- 
sensing elements of the image sensing device IMS are 
charged between time t 32 and time t 33 . The exposure time 
used this time is the exposure time t e2 , calculated in 
step S186 in Fig. 19, which is suitable for the medium 
luminance group GP2. 
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When the charging operation ends at time t 33 , the 
charges are transferred and read. At the same time, 
the value of the pixel shifting reference signal f in 
the yaw direction is changed to the initial value. 
Accordingly, the second lens group L2 is operated in 
the yaw direction on the basis of instruction values 
shown by the signal j. corresponding to the vibration 
signal h multiplied by the second coefficient. 

After a predetermined period of time has passed 
since time t 31 , the last timing pulse TP 4 is generated 
at time t 41 . Then, similarly to above, the photo- 
sensing elements of the image sensing device IMS are 
charged between time t 42 and time t 43 . The exposure time 
used this time is the exposure time t e3 , calculated in 
step S186 in Fig. 19, which is suitable for the low 
luminance group GP3 . 

When the charging operation ends at time t 43 , the 
charges are transferred and read. At the same time, 
the value of the pixel shifting reference signal e in 
the pitch direction is changed to the initial value. 
Accordingly, the second lens group L2 is operated in 
the pitch direction on the basis of instruction values 
shown by the signal i, corresponding to the vibration 
signal g multiplied by the second coefficient. 
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After the switch SW1 is turned off at time t 5 , the 
vibration detection and the operation of the second 
lens group L2 are stopped. 

A given point of an image formed on the image 
sensing device IMS while performing the image 
stabilization and the pixel shifting operation at time 
t 13L , t 21 , t 31 , t 41 and t 5 are at IM1, IM2, IM3, IM4 and 
IM1 in Fig, 4, respectively, thus image formation 
positions of the image are shifted by a half pixel 
distance from each other in the vertical and horizontal 
directions. In the four exposure operations of image 
sensing device IMS in the pixel shifting operation, 
exposure times suitable for respective luminance groups 
are used. 

Note, the reason for the pixel shifting reference 
waveform being a trapezoid shape rather than a square 
shape is to mitigate shock of sudden movement of the 
second lens group L2. 

Next, the principle for generating an image signal 
of a single high-resolution image by synthesizing a 
plurality of images obtained while performing pixel 
shifting operation, according to the fifth embodiment 
is explained with reference to Figs. 9A, 9B and 10A to 
10E. 

Figs. 9A and 9B are views for explaining relative 
position relationship between an image and the image 
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sensing device IMS in pixel shifting operation. Fig. 
9A corresponds to Fig. 4, and it shows that the 
position of the image shifts in the order of IM1, IM2, 
IM3, IM4, and IM1 with respect to the pixels of the 
image sensing device IMS fixed in the camera main body 
CMR. The above movement is equivalent to a case where 
the position of the image sensing device IMS moves in 
the order of IG1, IG2, IG3, IG4, and IG1 with respect 
to a fixed object. Since exposure time changes as the 
position of the image shifts in the pixel shifting 
operation, in other words, changes in accordance with 
the position of the image sensing device IMS, the 
lengths of the exposure times are expressed by the 
sizes of the circles in Fig. 9B. 

Next, a method for synthesizing a plurality of 
images obtained in the aforesaid operation is explained. 
An output signal from each pixel when the image sensing 
device IMS is at the position, IG1, be IGl(i, j ) , where 
i and j are coordinates of the pixel. The image 
sensing device IMS is an area sensor having m x n 
pixels. Similarly, let output signals when image 
sensing device IMS is at the positions, IG2, IG3 and 
IG4, be IG2(i, j), IG3(i, 3). and IG4(i, j), 
respectively. 

Figs. 10A to 10E show how to synthesize these four 
sets of image signals. 
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Let a new set of image signals representing 2m x 
2n pixels obtained by four sets of m x n pixels be 
denoted by IMG(u, v) . The four left uppermost corner 
pixels of the image signals IMG(u, v) are obtained by 
synthesizing the respective left uppermost corner 
pixels of the four original images as shown in Figs. 
10A to 10E. When a method for synthesizing images is 
considered with reference to Figs. 10A to 10E # it is 
possible to obtain an image signal of a single high- 
resolution image from the four original images by using 
the following four equations; 

IMG(u = 21-1. v = 2j) 

IGl(i, j) x K 2 + L 2 ...(16) 
IMG(u = 21-1, v = 2j-l) 

IG2(i, j) x K x + L x ...(17) 
IMG(u = 2i, v = 2j-l) 

163(1, j) x K 2 + L 2 ...(18) 
IMG(u - 2i, v = 2j) 

<- IG4(i, j) x K 3 + L 3 ..-(19) 



Here, coefficients K x to K 3 and offset values and 'L 1 to 
L 3 are for correcting differences in output levels of 
the four original images due to the difference in 
exposure time. Regarding K t to K 3 , they are represented 
as , 



-86- 



K, = C/t, 



el 



. . . (20) 



K 2 = C/t, 



(21) 



K 3 = C/t 



e3 



(22) 



where C is a constant; therefore, K x > K 2 > K 3 . 
Regarding the offset values L x to L 3 , they can be 
determined in accordance with the characteristics of 
the image sensing device IMS and luminous exposure. 

Fig. 23 is a graph showing the feature of the 
fifth embodiment. Similarly to Fig, 20 , the abscissa 
indicates luminance of the object and the ordinate 
indicates output voltage. Here, the coefficient K 3 for 
the image signals IG4(i, 3) which are obtained by 
exposing the image sensing device IMS for the longest 
period is set to 1, and the image signals IGl(i, 3) and 
IG3(i, 3) are synthesized after amplified by K 2 , 
further, the image signal IG2 is synthesized after 
amplified by K x . Further, referring to Fig. 23, the 
effective range of the output voltage of the original 
image signal is set between and V max , and the output 
voltage of the image signal IG4(i, 3) which is higher 
than the upper limit of the effective range V max , the 
output voltage of the image signals IGl(i, 3) and IG3(i, 
3 ) which are higher than the upper limit of the 
effective range and the lower than the lower limit 
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of the effective range V min , and the output voltage of 
the image signal IG4(i, j) which is lower than the 
lower limit of the effective range V min are not used for 
image synthesis. Regarding pixels corresponding to the 
image signals which are not used in the image synthesis, 
the missing pixels are interpolated on the basis of 
image information on the neighboring pixels after the 
four images are synthesized. 

More specifically, image signals obtained by 
exposing the image sensing device IMS for a short 
exposure time are amplified with a large gain, and 
image signals obtained by exposing the image sensing 
device IMS for a long exposure time are amplified with 
a small gain, thereby realizing normalization of the 
image signals for obtaining a synthesized high- 
resolution image. In this manner, in a case where 
pixels included in a given area in one of the plurality 
of images are saturated but pixels included in the same 
area of another image are not saturated, image signals 
corresponding to the saturated pixels are interpolated 
on the basis of image signals of the non- saturated 
pixels. With the aforesaid operation, it is possible 
to faithfully reproduce the image within a luminance 
range between log H3 and log H4* 

In the fifth embodiment as described above, among 
the four exposure operations performed in the pixel 
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shifting operation, the two exposure operations are 
performed using the same exposure time which is 
suitable for luminance of the main object. This is 
because more information on the main object can be 
obtained in this manner. 

Further, the signal normalization in image 
synthesis is controlled based on the exposure times in 
the fifth embodiment. Alternately, the signal 
normalization may be performed so as to control output 
values in the same area in the plurality of images 
become equal. 

Note, the aforesaid pixel shifting operation and 
the image synthesis method is used when image signals 
are obtained from a black-and-white image sensing 
device and a multiple-CCD type color image sensing 
device using a color separation prism. When image 
signals are obtained from a single CCD type color image 
sensing device covered with a mosaic color filter, 
although there are little differences in pixel shifting 
amount in the pixel shifting operation and the image 
synthesis method, the basic ideas of the pixel shifting 
operation and the image synthesis are the same. 

According to the fifth embodiment as described 
above , 

(1) It is possible to obtain an image of high 
resolution in wide dynamic range by changing luminous 
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exposures, used for sensing respective images obtained 
while performing pixel shifting operation, on the basis 
of luminance information, obtained by performing 
photometry by the sensors, on a plurality of divided 
areas of an image, and synthesizing the sensed images. 

(2) Since main object is inferred and luminous 
exposures are changed on the basis of the detected 
result, image reproductivity of the main object 
improves . 

(3) The more number of images are sensed using 
exposure time suitable for the luminance of the main 
object than the number of images sensed using any other 
exposure times, image reproductivity of the main object 
improves . 

< Sixth Embodiment > 

The number of the image formation positions 
shifted in the pixel shifting operation in the fifth 
embodiment is fixed, and only the luminous exposure is 
changed in accordance with the luminance of the main 
object. In a sixth embodiment as described below, the 
optimum pixel shifting mode is selected on the basis of 
image sensing conditions , and luminous exposures 
suitable for the selected pixel shifting mode are 
determined. 
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Fig. 24 is a flowchart showing control processing 
in the camera main body CMR according to the sixth 
embodiment; Figs. 12A and 12B are views for explaining 
relative position relationship between an image and the 
image sensing device IMS in pixel shifting operation 
according to the sixth embodiment; and Figs. 13A to 13C 
are views for explaining the principle of image 
synthesis in a second pixel shifting mode (will be 
explained later) . The sixth embodiment will be 
explained with reference to accompanying drawings. 

The control processing in the camera main body CMR 
in the sixth embodiment is basically the same as that 
shown in Fig. 18 explained in the fifth embodiment. 
However, the process performed in step S1115, i.e., "to 
set exposure and pixel shifting conditions" is realized 
by a sub-routine as shown in Fig. 24 # thereby the 
advantage of the sixth embodiment can be obtained. 
Below, the control processing according to the sixth 
embodiment will be explained with reference to Figs. 18 
and 24. Since the processes shown in Fig. 18 have been 
already explained in detail in the fifth embodiment, 
they are only briefly explained in the sixth embodiment. 

Referring to Fig. 18, when it is determined in 
step S102 that the switch SW1 is ON, then the process 
proceeds to step Sill. Thereafter, an IS start 
instruction is transmitted to the lens LNS in step Sill 
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and parameters are received from the lens LNS in step 
S112. Next in steps S113 and S114, photometry and 
focus state detection are performed, and the obtained 
results are transmitted to the microcomputer LCPU of 
the lens LNS. 

In the next step, S1115, the processes shown in 
Fig. 24 are performed. In step S261 in Fig. 24, an 
image formed on the sensor SNS is divided into a 
plurality of blocks in the same manner as performed in 
step S181 in Fig. 19, explained in the fifth embodiment. 

In the following step S262, the main object is 
inferred in the same manner as performed in step S182 
in Fig. 19. 

In step S263, the microcomputer CCPU requests 
transmission of the peak value of vibration angular 
velocity, co peaJC , in a predetermined period of time to 
the microcomputer LCPU. In turn, the microcomputer 
LCPU transmits the peak value of the vibration angular 
velocity o> peak occurring in a two second interval, for 
example, to the microcomputer CCPU. 

In step S264, the maximum vibration value 6 while 
exposing the image sensing device IMS when the IS 
function is not operated is calculated on the basis of 
the following equation. 



6 = f x co peak x t exp 



.(23) 
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where, f denotes a focal length of the optical system, 
and t exp is an exposure time, i.e., charging period, of 
the image sensing device determined on the basis of the 
5 result of photometry. The maximum vibration value 6 
obtained here is used in the subsequent steps as an 
index of vibration for determining whether or not pixel 
shifting operation should be performed. 

O In step S265, the value of the maximum vibration 

w 

y| 10 value 8 is checked. If the maximum vibration value 5 
•ug is equal or less than a predetermined value DELI, then 

ry it is determined that effect of vibration is small, 

3 

q thus the quality of an image would improve by 

performing pixel shifting operation. Accordingly, the 
15 process proceeds to step S271 and the number of image 
formation positions N SF , is set to four. Here, the 
number of image formation positions, four, indicates 
the same pixel shifting operation and image synthesis 
explained in the fifth embodiment. This overall 
20 operation is referred to as "first pixel shifting mode" 
in the sixth embodiment. When the first pixel shifting 
mode is decided, processes in steps S272 to S275 are 
performed. 

In steps S272 to S275, the same operations as 
25 those performed in steps S183 to S186 in Fig. 19 are 
performed, and exposure times t el to t en (n=3) to be 
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used in four exposure operations in the pixel shifting 
operation are determined. After step S275 # the process 
proceeds to step S266. 

Whereas, if it is determined in step S265 that the 
vibration value 5 is larger than the value DELI and 
equal or less than a predetermined value DEL 2 (DEL2 > 
DELI), then the process proceeds to step S281 where the 
number of image formation positions N SF is set to two. 
This is because deterioration of an image is expected 
to some degree even if image stabilization is performed. 
Therefore, a mode which requires less image formation 
positions is selected (this mode is referred to as 
"second pixel shifting mode") so as to reduce 
deterioration of the image quality due to vibration and 
to realize an optimum improvement in image quality. 
Details of the second pixel shifting mode are explained 
later . 

In step S282, a subordinate object which is the 
second important object is inferred. For instance, a 
block having the largest area among all the blocks, 
except the block including the main object, which are 
divided in step S261 is determined to include the 
subordinate object. 

In step S283, luminance of the block including the 
main object, determined in step S262, and luminance of 
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the block including the subordinate object , determined 
in step S282, are calculated. 

In step S284, luminous exposure which is 

suitable for the luminance of the main object 
(precisely, of the block including the main object) and 
luminous exposure E sub which is suitable for the 
luminance of the subordinate object (precisely, of the 
block including the subordinate object) are calculated 
on the basis of the luminances obtained in step S283. 

In step S285, exposure time t emain which is suitable 
for the main object and exposure time t esub which is 
suitable for the subordinate object are calculated on 
the basis of the luminous exposures and E sub , 

calculated in step S284. Thereafter, the process 
proceeds to step S266. Note, the second pixel shifting 
mode is explained later in detail. 

Further, in step S265, when it is determined that 
the vibration value 5 is greater than the predetermined 
value DEL2, then in step S291, the number of image 
formation positions N SF is set to 1 indicating that no 
pixel shifting operation is to be performed. The 
reason for setting the number of image formation 
positions N SF to 1 is that the effect of the vibration 
which causes deterioration of an image is stronger than 
the effect of pixel shifting operation which improves 
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the quality of the image. Therefore, the pixel 
shifting is disabled. 

In step S293, luminance of the block including the 
main object, determined in step S262, is calculated. 
5 In step S294, luminous exposure which is 

suitable for the luminance of the main object 
(precisely, of the block including the main object) is 
calculated on the basis of the luminance obtained in 
step S293. 

10 In step S295, exposure time t^^ which is suitable 

for the main object is calculated on the basis of the 
luminous exposure E^^ calculated in step S294. 
Thereafter, the process proceeds to step S266. 

After one of the series of processes in steps S271 

15 to S272, S281 to 285, and S291 to 295 are performed, 
the process proceeds to step S266. 

In step S266, information on the determined pixel 
shifting mode is transmitted to the microcomputer LCPU 
of the lens LNS. 

20 In step S267, the type of the pixel shifting mode 

is displayed on the display device DISP of the camera 
main body CMR to inform the user of which mode is used 
for photographing an image. 

After step S267 is completed, the process returns 

25 to step S116 in Fig. 18. 
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In step S116, determination of the state of the 
switch SW2 is performed, and if SW2 is ON, then the 
process proceeds to step S117. 

In steps S117 to S122, the pixel shifting is 
performed as explained in the fifth embodiment, however, 
in the sixth embodiment, the pixel shifting is 
performed in accordance with the selected pixel 
shifting mode determined in steps S271 to S295. 
Therefore, the pixel shifting operation is performed in 
the selected pixel shifting mode if either the first or 
second pixel shifting mode is selected. Whereas, if it 
is determined in the steps S261 to S267 not to perform 
pixel shifting operation, image formation position is 
not shifted. 

Then, in step S123, the completion of the pixel 
shifting operation (or completion of the storing of 
required image signals) is informed to the 
microcomputer LCPU, and the process proceeds to step 
S124. 

In step S124, the image synthesis processing 
corresponding to the selected pixel shifting mode is 
performed if the first or second pixel shifting mode is 
selected. If the pixel shifting operation was not 
performed, there is no need to perform image synthesis, 
therefore, the photographed image is directly outputted. 
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Then, in step S125, the obtained image is recorded 
and the process returns to step S102. 

The second pixel shifting mode, namely, a case 
where the number of image formation positions is two, 
is explained with reference to Figs, 12A and 12B. 

Fig. 12A shows shift of an image with respect to 
the image sensing device IMS. As shown in Fig. 12A, 
the image sensing device IMS is charged and read when a 
given point of an image is at a position IM21. Then, 
after the point of the image is shifted to the position 
IM22, the image sensing device is charged and read 
again. Thereafter, the image formation position is 
moved back to the initial position so that the point of 
the image is formed at the position IM21. 

Fig. 12B shows a movement of the image sensing 
device IMS equivalent to the aforesaid movement of the 
image shown in Fig. 12A. The image sensing device IMS 
is exposed for the exposure time t^^ when a pixel of 
the image sensing device IMS which is at an initial 
position IG21. Thereafter, the pixel of the image 
sensing device IMS is moved to the position IG22 by 
pixel shifting operation and the image sensing device 
IMS is exposed for the exposure time t esub , then moved 
back to the initial position IG21. 

Fig. 13 is a view for explaining an image 
synthesis method in the second pixel shifting mode. In 
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the second pixel shifting mode, on the basis of image 
signals of two images, IG21(i f j) and IG22(i, j), 
namely, image signals of (2 x m x n) pixels, an image 
IMG2(u, v) expressed with (4 x m x n) pixels is 
obtained. For this reason, pixels expressed by black 
dots in IMG2(u, v) are the same values as pixels in the 
images IG21(i, j) and IG22(i, j), and the pixels shown 
by white dots are interpolated with averages of the 
values of the neighboring four pixels (at edge, two or 
three pixels, instead of four pixels). 
In equations, 

IMG2(u=2i-l, v=2j) IG21(i, j) xK nBlll ...(24) 

IMG2(u=2i, v=2j-l) *- IG22(i, j) x K sub ,..(25) 
IMG2(u=2i-l, v=2 

<- {IG21(i, j) x + IG22(i, 3) * K sub + 

IG21(i, j-l) x Kjnain + IG22(i-l, j) x K sub }/4 ...(26) 
IMG2(u=2i, v=2j) 

<~ {IG21(i, j) x + IG22(i, 3) x K sub + 

IG21(1+1. j) x + IG22(i, j + l) x K sub }/4 ...(27) 

Here, and K sub are coefficients for correcting 

difference in output levels of the two original images 
due to the difference in exposure time, similarly to 
the first embodiment, and they are represented as, 
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K SU b = C/t esub 



.(28) 
• (29) 



According to the sixth embodiment as described 
above, in addition to the same effects as those of the 
first embodiment, 

(4) Since the optimum pixel shifting operation is 
selected in consideration of the vibration and the 
luminous exposure is changed so as to suit the selected 
pixel shifting mode, it is possible to obtain an image 
of optimum resolution in optimum dynamic range in 
accordance with the image sensing conditions. 

< Seventh Embodiment > 

In the sixth embodiment, the optimum pixel 
shifting mode is selected in accordance with vibration 
and the luminous exposure is controlled in accordance 
with the selected pixel shifting mode- In the seventh 
embodiment, whether or not the pixel shifting operation 
is to be performed is determined in accordance with an 
image sensing mode set by a user, and, if it is 
determined to perform pixel shifting operation, the 
pixel shifting mode as well as an luminous exposure are 
changed in accordance with the image sensing mode. 

The control processing in the camera main body CMR 
in the seventh embodiment is basically the same as that 
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shown in Fig. 18 explained in the fifth embodiment, 
similarly to the sixth embodiment. However, the 
process performed in step S1115, i.e., "to set exposure 
and pixel shifting conditions" is realized by a sub- 
routine as shown in Fig. 25, thereby the advantage of 
the seventh embodiment can be obtained. Below, the 
control processing according to the seventh embodiment 
will be explained with reference to Figs. 18 and 25. 
Since the processes shown in Fig. 18 have been already 
explained in detail in the fifth embodiment, they are 
briefly explained in the sixth embodiment. 

Referring to Fig. 18, when it is determined in 
step S102 that the switch SW1 is ON, the process 
proceeds to step Sill. Thereafter, an IS start 
instruction is transmitted to the lens LNS in step Sill 
and parameters are received from the lens LNS in step 
S112. Next in steps S113 and S114, photometry and 
focus state detection are performed, and the obtained 
results are transmitted to the microcomputer LCPU of 
the lens LNS. 

In the next step, S1115, the processes shown in 
Fig. 25 are performed. In step S361 in Fig. 24, an 
image formed on the sensor SNS is divided into a 
plurality of blocks in the same manner as performed in 
step S181 in Fig. 19, explained in the fifth embodiment. 



-101- 



In the following step S362 # the main object is 
inferred in the same manner as performed in step S182 
in Fig. 19 . 

In step S363 # the status of the image mode 
selection switch SWMOD (image sensing mode) provided in 
the camera main body CMR is determined, thereby image 
sensing conditions, such as exposure control mode, set 
by the user are determined. 

In step S364, whether or not the image sensing 
mode selected by the user is a landscape mode is 
determined. Landscape mode is an exposure control mode 
in which field depth is deepened by setting a small 
iris diaphragm control value (large F number) . When 
the landscape mode is set, it is expected that the 
object stands still, and the camera is held still, thus 
vibration would not occur in most cases. Accordingly, 
the process proceeds to step S371, where a "high- 
resolution" mode for sensing four images while shifting 
between four image formation positions in pixel 
shifting operation is set. When it is determined in 
step S364 that the set image sensing mode is not the 
landscape mode, then the process proceeds to step S365. 

In step S365, whether or not the selected image 
sensing mode is a portrait mode is determined. 
Portrait mode is an exposure control mode in which 
field depth is narrowed by setting the iris diaphragm 
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control value to near open (small F number). Since it 
is expected that the conditions for photographing in 
the portrait mode is similar to those of the landscape 
mode, the process proceeds to step S371. Whereas, if 
it is determined in step S365 that the set image 
sensing mode is not the portrait mode, then the process 
proceeds to step S366. 

In step S366, whether or not the set image sensing 
mode is a sport mode is determined. Sport mode is an 
exposure control mode for photographing a moving object 
as if it is not moving by shortening exposure time. 
When the sport mode is selected, it is expected that 
the object is moving and the camera may be panning* In 
other words, the movement of the camera similarly to 
the vibration by a large displacement amount is 
expected. Further, blurring of the object due to the 
movement of the object while performing pixel shifting 
operation is expected. Accordingly, improvement of 
image quality is not anticipated; on the contrary, the 
obtained image by performing pixel shifting operation 
would be unnatural. Therefore, when the sport mode is 
set, the process proceeds to step S391 where N SF is set 
to one and the pixel shifting is disabled. If it is 
determined in step S366 that the sport mode is not set, 
namely, when the set image sensing mode is not any of 
the landscape, portrait, and sport modes, the process 
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proceeds to step S381, and the number of image 
formation positions N SF is set to two. 

In steps S371 to S375, steps S381 to S385, and 
steps S391 to S395, the same processes as those 
performed in steps S271 to S275, steps S281 to S285, 
and steps S291 to S295 shown in Fig, 24 and explained 
in the sixth embodiment, respectively, are performed. 
In short, luminous exposures are determined on the 
basis of the selected exposure mode. Since these 
operations have been explained in detail in the sixth 
embodiment, the explanation of them is omitted. 

After one of the processes in steps S375, S385 and 
S195 is performed, the process proceeds to step S367. 

In step S367, information on the determined pixel 
shifting mode is transmitted to the microcomputer LCPU 
of the lens LNS. 

In step S368, the type of the pixel shifting mode 
is displayed on the display device DISP of the camera 
main body CMR to inform the user of which mode is used 
for photographing an image. 

After step S3 6 8 is completed, the process returns 
to step S116. 

In step S116, determination of the state of the 
switch SW2 is performed, and if SW2 is ON, then the 
process proceeds to step S117. 
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In steps S117 to S122, the pixel shifting is 
performed in accordance with the selected pixel 
shifting mode as explained in the sixth embodiment. 

Then, in step S123, the completion of the pixel 
shifting operation (or completion of the storing of 
required image signals) is informed to the 
microcomputer LCPU, and the process proceeds to step 
S124. In step S124, the image synthesis processing 
corresponding to the selected pixel shifting mode is 
performed as described in the sixth embodiment. 

In step S125, the obtained image is recorded and 
the process returns to step S102. 

According to the seventh embodiment as described 
above, in addition to the same effects as those of the 
fifth embodiment, 

(5) It is possible to perform an optimum pixel 
shifting operation suitable for movements of both a 
camera and an object by determining whether or not it 
is appropriate to perform pixel shifting operation and 
changing pixel shifting modes, in accordance with an 
image sensing mode set by the user. Accordingly, it is 
possible to obtain an image of optimum resolution in 
optimum dynamic range in accordance with the image 
sensing conditions. 

Further, an image sensing mode is selected on the 
basis of different photographing situations which 
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require different exposure control, however, an image 
sensing mode may be selected on the basis of the result 
of focus state detection. 

<Modif ications> 

In the fifth to seventh embodiments, a lens group 
in the optical system is moved in the orthogonal 
direction with respect to the optical axis of the 
optical system, thereby used as an image shifting means 
for realizing image stabilization and pixel shifting 
operation by utilizing optical shifting feature of the 
lens group. Alternately, it is possible to use a pair 
of transparent plates between which transparent liquid 
is filled, so-called, a variable apical angle prism is 
used. 

In addition, the features of the seventh 
embodiments are achieved regardless of the existence of 
image stabilization function. 

Further, exposure time is changed in order to 
change luminous exposures for sensing a plurality of 
images when performing pixel shifting operation. 
Instead, the iris diaphragm can be changed. 
Alternately, it is possible to change the luminous 
exposure by providing a transmitting light adjusting 
means, such as an electrochromic (EC) element and a 
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liquid crystal element, in the optical system and 
changing the density of the element. 

Further, the effects of the fifth to seventh 
embodiments can be obtained if a plurality of images 
are sensed in different exposure levels without 
shifting image formation position. In such a case, the 
resolution of an image is not improved; however, it is 
possible to widen a dynamic range in accordance with 
the state of an object. 

< Eighth Embodiment > 

The control processing performed by the 
microcomputer CCPU of the camera main body CMR 
according to an eighth embodiment is basically the same 
as that shown in Fig. 18, except the process in step 
S124. More specifically, in step S124 in the eighth 
embodiment, blurring is determined from image signals 
of a plurality of images obtained in pixel shifting 
operation, then the images are synthesized by a 
predetermined method, or may not be synthesized, in 
accordance with the determination result, and a single 
image of high resolution is obtained. The details will 
be explained later with reference to Fig. 33. 

Note, the control processing performed by the 
microcomputer LCPU of the lens LNS is substantially the 
same as that shown in Fig. 7. 
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Further, the control processing shown in Figs. 7 
and 18 are performed in the same timing as shown by the 
timing chart in Fig. 8 in the same manner as described 
in the first embodiment. 

Next, a method of determining blurring caused by 
vibration of the camera on the basis of a plurality of 
images sensed in pixel shifting operation will be 
explained with reference to Figs. 26A to 32. 

Figs. 26A and 26B, and Figs. 27A and 27B are 
graphs for explaining the principle for calculating the 
relative shifted amount between two images on the basis 
of correlation between image signals of the 
corresponding line of the two images. In these figures, 
the abscissa indicates coordinate of pixel of the image 
sensing device, and the ordinate indicates output 
signal value of each pixel. 

The curve IMC1 in Fig. 26A shows first image 
signals of a first image (referred to as "first image 
signals" hereinafter) obtained at a given time, and the 
curve IMC2 shows image signals of a second image 
(referred to as "second image signals" hereinafter) 
obtained at time after a predetermined period has 
passed since the first image is sensed. It is assumed 
that the two images are shifted from each other by a 
distance which is the integer multiple number of a 
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distance between pixels, due to pixel shifting 
operation or vibration. 

Fig. 26B shows a state in which the image signals 
IMC2 of the second image is shifted by two pixel 
distances to the left, and the first image signals and 
the second image signals exactly coincides. 

Here, the correlation CR between the first and 
second image signals is denoted by, 

CR = 1 - 2{ABS(IMC2(i) - IMC1 ( i) ) / ( IMC1 ( i) + 
IMC2(i) )} . . . (30) 

where ABS is an abbreviation of an absolute value, i is 
the pixel number, and 2 is a summation operator from i 
= 1 to a predetermined pixel number. In this case, 
when the shift amount, ST, of the image signals IMC2 is 
-2, the correlation CR becomes the maximum, 1. This 
state is expressed as STmax = -2, CRmax = 1. 

Fig. 27A shows a case where the shifted amount of 
the image signals of the two images is not an integer 
multiple number of the interval between pixels (1.5 
pixels in Fig. 27A) . 

Fig. 27B shows a state in which the image signals 
IMC2 of the second image is shifted by one pixel 
distance to the left, and the maximum correlation CRmax 
is 0.8, in this case. Even through the image signals 
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IMC2 is shifted by another pixel distance to the left, 
the maximum correction CRmax does not become 1. 

Fig. 28 shows the relationship between the shift 
amount ST and the correlation CR. 

Referring to Fig. 28 , when the correlation CR is 
calculated for shift amount ST between +2 and -3 
(integers only), the maximum correlation value CRmax is 
obtained when the shift amount ST is -1 and -2. 
Further, when the correlation CR is interpolated by 
linear regression as shown in Fig. 28, it is known that 
the correlation CR has a real maximum value of 0.9 when 
the shift amount ST is -1.5. Thus, STmax = -1.5 and 
CRmax = 0.9. 

The aforesaid calculation method is for obtaining 
shift amount between image signals of corresponding 
lines of the two images, however, it is possible to 
calculate shift amounts between image signals of two 
images in the up-and-down and right-and-left directions 
(namely, in two dimensions) by applying the aforesaid 
calculation method in two dimensions. More 
specifically, as disclosed in Japanese Patent 
Application Laid-Open No. 64-10787, image signals of 
two dimensional images , which are taken at different 
times, are sequentially compressed by projecting them 
on an abscissa and an ordinate, and a horizontal 
movement vector is calculated on the basis of a couple 



-110- 



Ill 



of the image signals projected on the abscissa and a 
vertical movement vector is calculated on the basis of 
a couple of the image signals projected on the ordinate. 
Alternately, correlation is calculated while 
5 sequentially shifting two dimensional images in two 

directions, thereby directly obtaining movement vectors 
in two dimensions. 

Fig. 29 is a drawing showing image formation 
positions when the precise pixel shifting is performed 

10 without being affected by vibration. Similarly to Fig. 
4, Fig. 29 shows image formation position of a given 
point of the image on the image sensing device IMS. 

In Fig. 29, in a state in which the given point of 
an image is at a position IM1 in the left -lower pixel, 

15 first image signals are obtained (charges the image 

sensing device IMS and read the charges from the image 
sensing device IMS). Then, the image formation 
position is shifted by a half pixel distance to the 
horizontal direction (corresponding to yaw direction) 

20 so as to move the given point to a position IM2, where 
second image signals are obtained. 

Fig. 30 is a graph showing the correlation CR 
between the first and second image signals which were 
obtained in the states explained with reference to Fig. 

25 29. The abscissa indicates shift amount between image 
signals of two images, and the shift amount in the 
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horizontal direction is denoted by STH, and the shift 
amount in the vertical direction is denoted by STV. 
The ordinate indicates correlation CR. A line CRH 
shows correlation while shifting relative position 
between the two images in the horizontal direction by 
the shift amount of STH (between -3 and +3 pixel 
distances), and a line CRV shows correlation while 
shifting relative position between the two images in 
the vertical direction by the shift amount of STV 
(between -3 and +3 pixel distances). 

Since the image formation position shown in Fig. 
29 is shifted by a half pixel in the horizontal 
direction, and is not shifted in the vertical direction 
(pitch direction), the shift amounts, STH and STV, with 
which the correlation CR reaches the maximum value 
(called "maximum correlation shift amounts" 
hereinafter) are STHmax = -0.5 and STVmax = 0. The 
maximum correlation values CRHmax and CRVmax in this 
case are little less than 1. This is because the 
second image is obtained by shifting the image 
formation position by a half -pixel distance, not by the 
integer multiple number of the interval between pixels; 
therefore, points of the first image sensed by 
respective pixels of the image sensing device IMS are 
not formed on the pixels when sensing the second image. 
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The same holds true for correlation in the vertical 
direction, therefore, CRVmax does not become 1. 

Fig. 31 is a view showing a case where a large 
vibration occurs while performing pixel shifting 
operation and the vibration can not be completely 
compensated even through image stabilization is 
functioning . 

Referring to Fig, 31, in a state in which a given 
point of an image is at a position IM1 in the left- 
lower pixel, first image signals are obtained. Next, 
the image formation position is to be ideally shifted 
by a half pixel distance to the horizontal direction as 
indicated by a broken arrow, however, due to the 
vibration, it is assumed that the given point is 
shifted to a position IM2, and second image signals are 
obtained. 

Fig. 32 is a graph showing correlation CR between 
the first and second image signals obtained at 
different image formation positions as described with 
reference to Fig. 31. References used in Fig. 32 are 
the same as those used in Fig. 30, 

Referring to Fig. 32, the maximum shift amounts in 
the horizontal and vertical directions, STHmax and 
STVmax, are -0.7 and -0.3, respectively. Let the 
differences between these maximum shift amounts and 
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maximum shift amounts obtained when no vibration occurs 
as shown in Fig. 30 be 5 H and 5 V# then 



5 H = ABS(-0.7 - (-0.5)) = 0.2 
^ = ABS(-0.3 - 0) = 0.3 



(31) 
(32) 



6 H and 6 V are blurring amounts (unit is "pixel 
distance"), caused by vibration, in the horizontal and 
vertical directions and ABS is an abbreviation of 
"absolute value". When these blurring amounts 6 H and b v 
exceed a predetermined value, e.g., 6^ = ±0.1 pixel 
distance, improvement in resolution by synthesizing the 
first image signals and second image signals can not be 
expected. In such a case, pixel shifting mode, for 
example, is changed. 

Here, since the image stabilization is functioning 
in the eighth embodiment, blurring due to vibration 
ideally does not occur; however, there is a possibility 
that blurring may occur due to an output error of the 
vibration sensors GRP and GRY and due to a large 
vibration which exceeds the ability of the image 
stabilization system of the camera. Blurring caused by 
the aforesaid reasons is uniformly observed in the 
entire area between sensed images. Therefore, by 
performing the aforesaid correlation calculation over 
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the entire area of the sensed images, it is possible to 
determine blurring due to vibration. 

The principle for generating a single high- 
resolution image by synthesizing a plurality of images 
obtained while performing pixel shifting operation is 
same as that explained in the first embodiment with 
reference to Figs. 9A to 10E and equations (7) to (10), 
therefore, explanation of it is omitted. 

Note, the aforesaid pixel shifting operation and 
the image synthesis method is used when image signals 
are obtained from a black-and-white image sensing 
device and a multiple-CCD type color image sensing 
device using a color separation prism. When image 
signals are obtained from a single CCD type color image 
sensing device covered with a mosaic color filter, 
although there are little differences in pixel shifting 
amount in the pixel shifting operation and the image 
synthesis method, the basic ideas of the pixel shifting 
operation and the image synthesis are the same. 

Fig. 33 is a flowchart of determination processing 
on vibration and image synthesis operation, as 
explained with reference to Figs. 26A to 32 and 9A to 
10E, performed during pixel shifting operation. This 
processing corresponds to the process performed in step 
S124 in Fig. 18. 
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First in step S161, a counter CNT for counting the 
number of image formation positions is initialized to 0. 

In step S162, correlation between the first and 
second image signals are calculated on the basis of the 
equation (30). The process proceeds to step S163 where 
the maximum shift amounts in the horizontal and 
vertical directions, STHmax and STVmax, are calculated 
on the basis of the result of the correlation 
calculated in step S162. 

Thereafter, in step S164, the maximum correlation 
in the horizontal and vertical directions, CRHmax and 
CRVmax are calculated. Further, in step S165, the 
blurring amounts in the horizontal and vertical 
directions, 5 H and 5 V , are calculated. 

In step S166, the counter CNT is increased by 1, 
and in step S167, the value of the counter CNT and the 
predetermined value N SF indicating the number of image 
formation positions are compared. For example, if N SF 
is 4, calculations performed in steps S162 to S165 are 
performed three times, i.e., between the first and 
second image signals, the second and third image 
signals, and the third and fourth image signals. 
Therefore, if CNT is less than N SF -1, then the process 
returns to step S162, and the processes in steps S162 
to S166 are performed on the next pair of image signals. 
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When the CNT reaches N SF -1, the process proceeds to 
step S168. 

In step S168, the respective blurring amounts 5 H 
and by which are calculated in step S165 are compared 
5 to the predetermined value b^. If both of the 
blurring amounts 5 H and 5 V are less than the 
predetermined value 8 MAX# improvement in resolution is 
expected by synthesizing a plurality of images obtained 
in pixel shifting operation, therefore, the process 

10 proceeds to step S169 where image synthesis is 
performed in the same manner as described with 
reference to Figs, 10A to 10E. 

In step S170, a message indicating that a normal 
image synthesis of a plurality of images obtained while 

15 performing pixel shifting operation is performed is 
displayed on the display DISP, Thereafter, in step 
S171, the process returns to the main processing shown 
in Fig. 18. 

Whereas, if it is determined in step S168 that at 
20 least one of the blurring amounts 5 H and 5 V is equal or 
greater than the predetermined value 5^, then the 
process goes to step S172 where a warning message 
notifying that image synthesis of the plurality of 
images obtained while performing pixel shifting 
25 operation is not performed is displayed on the display 
DISP. Thereafter, the process returns to the main 
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processing shown in Fig. 18 in step S171. In the above 
case, one of the plurality of images, not a synthesized 
image, such as the one represented by the first image 
signals, is output ted. 

According to the eighth embodiment as described 
above , 

(1) Blurring due to vibration is determined on 
the basis of correlation between image signals of a 
plurality of images which are obtained in pixel 
shifting operation, and whether or not image synthesis 
of the plurality of images should be performed is 
determined in accordance with the degree of the 
blurring. Accordingly, deterioration of image quality 
due to vibration can be reduced, and a high-resolution 
image obtained by synthesizing a plurality of images 
obtained while performing pixel shifting operation can 
be generated. 

(2) Since whether or not image synthesis of the 
plurality of images was performed is notified to the 
user using a display, the user can check the resolution 
of an obtained image. Further, in a case where image 
synthesis is not performed because of strong effect of 
vibration, it is possible for the user to perform image 
sensing operation once more. 

(3) Blurring amounts are calculated each time an 
image formation position shifts while performing pixel 
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shifting operation, therefore, blurring is detected at 
high precision. Accordingly, deterioration of image 
quality by performing pixel shifting operation due to 
vibration is prevented. 

<Ninth Embodiment > 

In the eighth embodiment, blurring is determined 
each time the image formation position is shifted in 
the pixel shifting operation, and when large vibration 
occurs even one time, the image synthesis is not 
performed. In the following ninth embodiment, image 
signals of one more image are obtained in the initial 
image formation position of pixel shifting operation, 
and the image signals are used for blurring detection. 

Fig. 34 shows a track of a given point of an image 
formed on the image sensing device IMS when performing 
pixel shifting operation. Figs. 35 and 36 are 
flowcharts of control processing in the camera main 
body according to the ninth embodiment. Explanation on 
operation according to the ninth embodiment follows 
with reference to Figs. 34 to 36. 

Referring to Fig. 34, an image is to be ideally 
shifted on the track, shown by a broken line, with 
respect to the image sensing device IMS , and image 
signals of five images (respectively referred to as 
"first to fifth image signals" hereinafter) are 
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obtained. However, it is assumed that the actual image 
formation position shifts on the track, shown by a 
solid line, in the order of IM1, IM2, IM3, IM4 and IM5. 
In order to increase resolution in pixel shifting 
operation, the first to fourth image signals are 
synthesized, and the first and fifth image signals are 
used to calculate correction for determining blurring 
in the images. More specifically, if no blurring is 
determined, the first image signals obtained at the 
position IM1 and the fifth image signals obtained at 
the position IM5 should completely match, and STHmax 
and STVmax should be 0, 5 H and 5 V should be 0, and 
CRHmax and CRVmax should be 1. Whereas, if images are 
affected by vibration, blurring amounts in the 
horizontal and vertical directions 5 H and 6 V have values 
other than 0 which represent resultant blurring 
occurred in pixel shifting operation. 

The control processing shown in Fig. 35 is 
basically the same as that explained in the eighth 
embodiment with reference to Fig. 18; however, step 
S122 in Fig. 18 where the number of image formation 
positions is compared to a predetermined value is 
different. More specifically, in step S122 in Fig. 18, 
after image signals of four images are obtained, the 
process proceeds to step S123. 
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Whereas, in the ninth embodiment as shown in Fig. 
35 , the image formation position is returned to the 
initial position of the pixel shifting operation after 
image signals of four images are obtained, and a fifth 
image to be used for determining blurring of images is 
sensed. Accordingly, after the counter CNT for 
counting the number of image formation positions 
becomes N SF +1, e.g., 5, in step S1122, the process 
proceeds to step S123. 

After the process in step S123 is performed, a 
plurality of images are synthesized in step S124 in 
such a manner as shown in the flowchart in Fig. 36. 
The steps other than steps S1122, and S124 are the same 
as that explained in the eighth embodiment with 
reference to Fig. 18, therefore, explanation of them 
are omitted. 

Fig. 36 is a flowchart showing control processing 
of image synthesis performed in step S124 in Fig. 35. 
Similarly to the processing shown in Fig. 33 explained 
in the eighth embodiment, detection of blurring 
occurred while performing pixel shifting operation and 
image synthesis are performed. 

In step S461, correlation between the first and 
fifth image signals is calculated on the basis of the 
equation (30) . 
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Thereafter, in step S462, the maximum shift 
amounts STHmax and STVmax are calculated on the basis 
of the correlation result. 

In step S463, the maximum correlation CRHmax and 
5 CRVmax are calculated. 

In step S464, the blurring amounts 5 H and 6 V are 
calculated. In the ninth embodiment, the maximum shift 
amounts STHmax and STVmax are 5 H and by. 

Then, in step S465, the respective blurring 
10 amounts 6 H and 6 V calculated in step S464 are compared 

to a first predetermined value 5 X . If both the blurring 
amounts 5 H and 6 V calculated in step S464 are smaller 
than the first predetermined value 6 1# it is determined 
that blurring due to vibration is very small and 
15 improvement in resolution is expected by synthesizing 
all the four images sensed while performing pixel 
shifting operation. Therefore, the process proceeds to 
step S466 where the four images are synthesized in the 
same manner as explained with reference to Figs. 10A to 
20 10E. 

In step S467, a message indicating that four 
images are synthesized is displayed on the display DISP. 
Then, the process proceeds to step S472 where the 
process returns to the control processing shown in Fig. 
25 35. 
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If it is determined in step S465 that at least one 
of the blurring amounts 6 H and 5 V is equal or greater 
than the first predetermined value b lt then the process 
proceeds to step S468. In step S468, the blurring 
amounts 8 H and 8 V are compared to a second predetermined 
value 8 2 which is greater than 6 X . If both of the 
blurring amounts 5 H and 8 V are smaller than the second 
predetermined value 6 2 , since improvement in resolution 
is expected by synthesizing a plurality of images 
although a synthesized image would somewhat blur 
because of vibration. Accordingly, the process 
proceeds to step S469 where two images out of four 
images are synthesized. This is because synthesis of a 
plurality of images increase resolution, however, there 
is a possibility that the quality of the synthesized 
image would decrease, because of effect of vibration, 
by synthesizing a plurality of images which are 
obtained in a long period of time. There is even a 
possibility that the quality of the synthesized image 
would be lower than the quality of each of the four 
images. Accordingly, in the ninth embodiment, all the 
four images are not synthesized to remove the effect of 
vibration, although the possible highest resolution can 
not be achieved. More specifically, the first and 
second image signals may be synthesized to increase the 
resolution in the horizontal direction, or the first 
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and third image signals may be synthesized, for 
instance. In these cases, pixels which are not 
provided with information from the synthesized image 
signals of the two images may be interpolated with 
their neighbor pixel values . 

After the image synthesis in a predetermined 
manner is completed in step S469, the process proceeds 
to step S470 where a message indicating that two images 
are synthesized is displayed on the display DISP. 
Thereafter, the process proceeds to step S472 and 
returns to the control processing shown in Fig. 35. 

Whereas, if at least one of the blurring amounts 
6 H and 5 V is larger than the second predetermined value 
5 2 , it is expected that the image would deteriorate by 
synthesizing a plurality of images because of large 
vibration, therefore, the process proceeds to step S471 
where a warning message indicating that image synthesis 
was not performed is displayed on the display DISP. 
Then, in step S472, the process returns to the control 
processing shown in Fig. 35. In this case, one of the 
first to fourth image signals, not synthesized image 
signals, e.g., the first image signals, are outputted 
as the final image signals. 

According to the ninth embodiment as described 
above, in addition to the same effects as those of the 
eighth embodiment, 
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(4) Blurring occurred while performing the pixel 
shifting operation is determined on the basis of two 
images, therefore, it takes a short time to determine 
the blurring. 

(5) Since an image which is first sensed in pixel 
shifting operation and an image sensed after an image 
formation position in the image shifting operation is 
moved back to the initial position are used for 
calculating correlation, the shifted amounts between 
the two images , calculated on the basis of the result 
of correlation, corresponds to blurring amounts* 
Therefore, calculation on the blurring amounts is 
simplified, since it is unnecessary to calculate 
correlation in consideration of shifted amounts due to 
the pixel shifting operation. 

(6) An optimum image synthesis mode is selected 
from a plurality of image synthesis modes in accordance 
with the blurring amounts; therefore, it is possible to 
prevent deterioration of an image caused by 
synthesizing a plurality of images obtained in pixel 
shifting operation, due to vibration. 

(7) Since the selected pixel shifting mode is 
displayed, the user can determine the resolution of an 
obtained image. 

< Tenth Embodiment > 
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In the eighth and ninth embodiments, blurring 
amounts are calculated on the basis of correlation 
between a plurality of images obtained while performing 
pixel shifting operation, and when at least one the 
5 blurring amounts is greater than a predetermined value, 
synthesis is disabled. In the following tenth 
embodiment, by providing means for correcting blurring, 
synthesis of a plurality of images is performed even 
when the blurring amount is greater than the 
10 predetermined value. 

Fig. 37 is an explanatory view showing a track of 
a given point of an image on the image sensing device 
IMS according to the tenth embodiment of the present 
yk invention. Fig. 38 is a graph showing correlation CR 

fy 15 between first and second image signals, and Fig. 39 is 
a flowchart of control processing in the camera main 
body. The operation of the tenth embodiment follow 
with reference to Figs. 37 to 39. 

Referring to Fig. 37, the given point of the image 
20 is to be ideally shifted by a half pixel distance in 

the horizontal direction, as indicated by a broken line. 
However, it is assumed that the position of the given 
point of the image is actually shifted from IM1 to IM2 , 
as shown by a solid line, because of vibration. 
25 Fig. 38 is a graph showing correlation between the 

two images sensed when the given point of the image is 
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at the positions IM1 and IM2. For convenience, the 
image sensed when the given point of the image is at 
the position IM1 is referred to as "first image IM1" , 
and the other image is referred to as "second image 
IM2" below. Referring to Fig. 38, it is known that 
STHmax is -1.5, STVmax is 0, 6 H is 1 (=ABS(-1.5 - (- 
0.5))), 8v is 0 (=ABS(0 - 0)), and CRHmax and CRVmax 
are 0.94. Here, the blurring amount in the vertical 
direction 5 V is 0, thus does not cause any problem; 
however, since the blurring amount 6 H in the horizontal 
direction corresponds to one pixel distance, if a 
plurality of images are synthesized in this state, the 
obtained image would have lower image quality than that 
of each of the plurality of images. However, in a case 
where a blurring amount between the two images is about 
the integer multiple numbers of pixel distance, by 
synthesizing the two images after one of these images 
is shifted in a direction, which is opposite to the 
blurring direction, by the distance of the integer 
multiple numbers of one pixel distance, it is possible 
to cancel out the blurring. For example, in Fig. 37, 
the second image IM2 is shifted by 1.5 pixel distances 
in the horizontal direction with respect to the first 
image IM1 due to pixel shifting operation and vibration. 
Thus, by synthesizing the first image IM1 and the 
second image IM2 after the second image IM2 is shifted 
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by one pixel distance, blurring due to vibration is 
canceled, and only the shift amount of a half pixel 
distance due to pixel shifting operation remains. 
More specifically , 



n H = INT (8^) 
N v = INT(8 V ) 



(33) 
(34) 



where INT() indicates the rounding off, to the nearest 
whole number, to change the blurring amounts 5 H and 5 V 
to integers n H and n v . Further, coordinate of pixels, i 
and j, of the image signals IG2(i, j ) described in the 
aforesaid equation (8) are changed on the basis of. 



i + n„ 



3 *- j + n v 



• . • (35) 
. • . (36) 



Accordingly, coordinates of pixels of the second image 
IM2 are shifted by a distance of the blurring amount 
before being synthesized with the first image IM1, 
thereby it is possible to cancel out the blurring. 

The aforesaid operation is also performed on the 
image signals IM3 and IM4 (i.e., IG3(i, j ) and IG4(i, 
j) of the equations (9) and (10)). In this case, 
correlation used for calculating blurring amounts is 
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preferably performed with respect to the first image 
IM1. 

By synthesizing images using the equations (7) to 
(10) after the aforesaid operation is performed on each 
of the plurality of images, it is possible to increase 
the resolution of an image by synthesizing the 
plurality of images obtained while performing pixel 
shifting operation even in a case where blurring which 
is greater than one pixel distance occurred between the 
plurality of images. 

The control processing performed by the 
microcomputer CCPU of the camera main body CMR 
according to the tenth embodiment is basically the same 
as that shown in Fig. 18 explained in the eighth 
embodiment, except a process in step S124. In step 
S124 in the tenth embodiment, processing as described 
in the flowchart in Fig. 39 is performed. 

First in step S561, a counter CNT for counting the 
number of image formation positions is initialized to 0. 

In step S562, correlation between the first and 
second image signals is calculated on the basis of the 
equation (30) . 

Thereafter, in step S563, the maximum shift 
amounts STHmax and STVmax are calculated on the basis 
of the correlation result. 
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In step S564, the maximum correlation CRHmax and 
CRVmax are calculated. 

In step S565, the blurring amounts 6 H and 5 V are 
calculated, 

5 In step S566, the blurring amounts 6 H and 5 V are 

rounded into integers (n H and n v ) . 

In step S567, the rounded blurring amounts n H and 
n v are evaluated, and if both of them are smaller than 
jk five pixel distances, then the process proceeds to step 

S 10 S568. Whereas, if at least one of the rounded blurring 

: lf amounts n H and n v is equal or greater than five pixel 

As : 

f!: distances, since there is a more possibility that each 

^ of the plurality of images blurs, a warning message 

.H indicating that blurring due to vibration is large is 

?! 

fU 15 displayed on the display DISP in step S575. 

O 

fU Thereafter, the process proceeds to step S574 and 

returns to the control processing in Fig. 18. In this 
case, synthesis of images is not performed, and one of 
the images sensed while performing pixel shifting 

20 operation is recorded on a recording device in step 
SI 2 5 in the processing shown in Fig. 18. 

Whereas, when the process proceeds from step S567 
to step S568, whether or not both of the maximum 
correlation values CRHmax and CRVmax are greater than a 

25 predetermined value, e.g., 0.5, is determined. If both 
of the maximum correlation values CRHmax and CRVmax are 
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greater than 0.5, the process proceeds to step S569. 
Whereas, if at least one of the maximum correlation 
values CRHmax and CRVmax does not exceed 0.5, then it 
is determined that reliability of the blurring amounts 
8 H and 5 V , calculated from the correlation, is low. If 
blurring is corrected on the basis of an incorrect 
result of blurring determination and the images are 
synthesized, the synthesized image would have lower 
quality than each of the original images. Accordingly, 
when at least one of the maximum correlation values 
CRHmax and CRVmax is equal or less than 0.5, the 
process proceeds to step S575 where a warning message 
is displayed and the process returns to the processing 
shown in Fig. 18, in step S574. 

If the integer blurring amounts n H and n v are 
smaller than the predetermined value (5, in this case) 
in step S567 and the maximum correlation values CRHmax 
and CRVmax are greater than the predetermined value 
(0.5, in this case) in step S568, then the process 
proceeds to step S569. 

In step S569, blurring correction is performed in 
accordance with the equations (35) and (36). 

In step S570, the counter CNT is increased by 1. 

Next in step S571, the value of the counter CNT is 
compared with a predetermined value N SF indicating the 
number of image formation positions. For example, when 
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N SF is 4, calculations performed in steps S562 to S569 
are performed three times between the first and second 
image signals, the second and third image signals, and 
the third and fourth image signals- Therefore, if CNT 
5 is less than N SF -l f then the process returns to step 
S562, and the processes in steps S562 to S569 are 
performed on the next pair of image signals. When the 
CNT reaches N SF -1, the process proceeds to step S572. 
Thereafter, in step S572, image synthesis is 

10 performed in the same manner as described with 
reference to Figs. 10A to 10E. 

In step S573, a message indicating that image 
synthesis is performed after correcting blurring is 
displayed on the display DISP. Then, in step S574, the 

15 process returns to the processing shown in Fig. 18. 

According to the tenth embodiment as described 
above, in addition to the same effects as those of the 
eighth embodiment, 

(8) Since synthesis of a plurality of images 

20 sensed while performing pixel shifting operation is 
performed after detecting and correcting blurring 
between the plurality of images, it is possible to 
obtain a high-resolution image by synthesizing the 
plurality of images even in a case where blurring 

25 amount, due to vibration, is larger than a pixel 
distance. 
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(9) Since whether or not to perform synthesis of 
a plurality of images obtained while performing pixel 
shifting operation is determined in accordance with the 
correlation between the plurality of images, it is 
possible to prevent deterioration of an image when 
pixel shifting operation was unsuccessful. 

Modifications > 

The above eighth to tenth embodiments can be 
applied to an image sensing apparatus which senses a 
plurality of images in a predetermined period and 
synthesizes the sensed images for purposes other than 
pixel shifting operation. For example, by applying the 
eighth to tenth embodiments to an image sensing 
apparatus which senses a plurality of images while 
changing luminous exposure and synthesizes the 
plurality of images in order to widen the dynamic range 
of a synthesized image, it is possible to obtain an 
image of a wide dynamic range without being affected by 
vibration. 

In another example, by applying the eighth to 
tenth embodiments to an image sensing apparatus which 
senses a plurality of images of the same scene at 
different timing to realize an effect of multiple 
exposures, plural number of a moving object are 
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recorded overlaid in a single image, while a still 
object is reproduced without blurring. 

< Eleventh Embodiment > 

Figs. 40 and 7 are flowcharts of control 
processing by the microcomputer CCPU of the camera main 
body CMR and the microcomputer LCPU of the lens LNS, 
respectively, according to an eleventh embodiment of 
the present invention. 

First, a flow of the control processing by the 
microcomputer CCPU of the camera main body CMR is 
explained with reference to Figs. 1 and 40. 

When the main switch (power switch) SWMN of the 
camera main body CMR is turned on in step S101, 
electric power is supplied to the microcomputer CCPU, 
then the process proceeds to step S102 where operation 
of the camera starts. 

In step S102, the state of the switch SW1, which 
is turned on in response to the half stroke of the 
release button, is detected. If the SW1 is off, then 
the process proceeds to step S103, where an instruction 
to stop image stabilization (IS) operation (IS stop 
instruction) is transmitted to the lens LNS. 

The steps S102 and S103 are repeatedly performed 
until the switch SW1 is turned on or the main switch 
SWMN is turned off. 
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When the switch SW1 is turned on in step S102, the 
process proceeds to step Sill. In step Sill, the 
microcomputer CCPU transmits an instruction to start IS 
operation (IS start instruction) to the microcomputer 
LCPU via the signal line DCL. 

Next in step S112, communication for obtaining 
parameters which are specific to the lens, such as F 
number and focal length of the lens, from the 
microcomputer LCPU is performed. 

Then, in step S113, the luminance of the object is 
sensed by the sensor SNS, and the charging period of 
the image sensing device for obtaining image signals 
and the value for controlling the iris diaphragm are 
calculated in accordance with the predetermined 
exposure control program, and the microcomputer CCPU 
transmits the calculation results to the microcomputer 
LCPU. 

The process proceeds to step S114 where the focus 
state is detected by the sensor SNS, and an instruction 
for operating the focus lens is transmitted to the 
microcomputer LCPU. 

In step S1115, the state of the pixel shifting 
mode selection switch SWSF is detected. Further, pixel 
shifting conditions, such as, whether or not the pixel 
shifting is to be performed and the number of image 
formation positions N SF on the image sensing device IMS 
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in an image sensing operation (if it determined not to 
perform pixel shifting, N SF is set to 1, whereas if it 
is determined to perform pixel shifting, then N SF is 
set to at least 2), are set on the basis of the result 
5 of the photometry. 

Thereafter, the process proceeds to step S116, 
where the state of the switch SW2 which is turned on in 
response to the full stroke of the release button is 
jZ detected. If the switch SW2 is OFF , then the process 

10 returns to step Sill, and steps Sill to S115 are 
CI repeated. Whereas, if it is detected that the switch 

M; SW2 is ON, then the process proceeds to step S117. 

* In step S117, a counter CNT for counting the 

number of image formation positions is initialized to 0 . 
m 15 Then, in step S118, a timing pulse which is a 

yy trigger signal for image sensing operation is generated, 

and transmitted to the microcomputer LCPU. 

In step S119, the microcomputer CCPU controls the 
image sensing device IMS, via the driver IMDR, to 
20 charge, then transfer the charges in the image sensing 
device IMS. 

In step S120, the image signals read at step S119 
are temporarily stored in the RAM of the microcomputer 
CCPU. 

25 In step S121, the counter CNT 1 is increased by 1. 
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In step S122, whether or not the counter CNT 
reaches the number of image formation positions N SF is 
determined. If it is not, then the process returns to 
step SI 18 and waits for the next timing pulse being 
generated, then pixel shifting operation is continued. 
If it is determined that the counter CNT has reached 
the number of image formation positions N SF , then the 
process proceeds to step S123. 

In step S123, completion of pixel shifting 
operation (or completion of the storing of required 
image signals) is informed to the microcomputer LCPU. 

Thereafter, the process proceeds to step S126, 
where blurring in images due to vibration is determined 
from the plurality of images obtained while performing 
pixel shifting operation. Details will be explained 
later with reference to Fig. 46. 

In step S127, blurring in images due to movement 
of an object is also determined from the plurality of 
images obtained while performing the pixel shifting 
operation. Details will be explained later with 
reference to Fig. 47. 

Then the process proceeds to step S128 where a 
plurality of images are synthesized to generate a 
single image in accordance with the result of blurring 
due to vibration and movement of the object, detected 
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in steps S126 and S127. Details will be explained with 
reference to Fig. 48. 

In step S129, the image outputted in step S128 is 
stored in the memory MEM. 

Accordingly, an image sensing operation is 
completed and the process returns to step S102. If the 
switch SW1 becomes ON in step S102, then the processes 
in step Sill and the subsequent steps are repeated, 
whereas, if the switch SW1 is OFF, then an instruction 
to stop IS operation is transmitted to the 
microcomputer LCPU in step S103. 

Control processing performed by the microcomputer 
LCPU of the lens LNS is the same as that explained in 
the first embodiment with reference to Fig. 7, 
therefore, the explanation of it is omitted. 

Further, method for determining blurring due to 
vibration from the plurality of images obtained while 
performing pixel shifting operation is the same as that 
explained in the eighth embodiment with reference to 
Figs. 26A to 32, the explanation of it is also omitted. 

Next, method for determining blurring in images 
due to movement of an object is explained with 
reference to Figs. 41 and 42. 

Fig. 41 is an explanatory view showing an image of 
an object formed on the photo -sensing surface of the 
image sensing device IMS. In the middle portion of the 
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image, the main object OBM, e.g., a person, is formed, 
and in the back of the main object, a subordinate 
object OBS is formed, further, in the back of the 
subordinate object OBS, a background BKG is formed. 
5 Here, it is assumed that a part of the person, e.g., 
the arm, is moving in the direction of the arrow. 

Fig. 42 is an explanatory view showing an image 
obtained by synthesizing four images of the object 
which is shown in Fig. 41 sensed while performing pixel 

10 shifting operation, without correcting blurring of the 
object. In the pixel shifting operation, the same 
exposure operation as that of multiple exposure 
operation is performed, and blurring due to movement of 
the object can not be corrected by compensating 

15 blurring due to vibration, which uniformly occurs over 
the images. In this case, therefore, the person 
reproduced in the synthesized image looks as if he/she 
has transparent four arms through which the subordinate 
object is seen, as shown in Fig. 42. Thus, the 

20 obtained image would not be a normal photograph which 
the user would have intended. 

Fig. 43 is a view for explaining method for 
dividing the photo- sensing surface of the image" sensing 
device IMS in order to detect partial blurring due to 

25 movement of an object as described above. In Fig. 43, 
the photo -sensing surface is divided into a plurality 
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of small areas AR(k) of, e.g., 216 (= 18 x 12) (k = 1 
to K, K = 216). Each divided area AR(k) consists of a 
plurality of pixels. On each divided area AR(k) , 
correlation and blurring amounts are calculated in such 
a manner as explained in the eighth embodiment with 
reference to Figs. 26A to 32. Then, if blurring 
amounts in all of the respective divided areas AR(k) 
are near zero, then it is determined that neither 
blurring due to vibration nor blurring due to movement 
of the object occurred. If blurring amounts in all of 
the respective divided areas AR(k) have almost the same 
value which is over a predetermined value, then it is 
determined that blurring due to vibration occurred. 
Further, if blurring amounts are large in specific 
divided areas, then it is determined that blurring due 
to movement of the object occurred. 

Fig. 44 is an explanatory view showing a 
determined result of blurring due to movement of the 
object performed on the basis of a plurality of images 
of the object, which is shown in Fig. 41, sensed while 
performing pixel shifting operation. In Fig. 44, an 
area ARMV, including the moving arm and enclosed by a 
solid line, has large blurring amounts. 

Fig. 45 shows an image obtained by synthesizing 
the plurality of images sensed in pixel shifting 
operation in accordance with the detected result of 
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blurring. Referring to Fig. 45, the area outside of 
the area ARMV (same as that shown in Fig. 44) i s 
obtained by synthesizing the image signals of four 
images obtained in pixel shifting operation to generate 
a high-resolution image. Whereas, image signals of the 
first image, out of the four images, is used for 
reproducing the area ARMV. As a result, the resolution 
of the image inside of the area ARMV is not improved, 
however, the undesired effect caused by exposing the 
moving object a plurality of times is prevented, as 
well as the resolution of the image in the area outside 
of the area ARMV is improved. 

As for method for generating a single high- 
resolution image by synthesizing a plurality of images 
obtained while performing pixel shifting operation is 
basically the same as that explained in the first 
embodiment with reference to Figs. 9A to 10E using 
equations (7) to (10). 

The above image synthesis method is for a case 
where no blurring due to vibration nor movement of an 
object occurs and resolution of the synthesized image 
can be improved in whole area. In the eleventh 
embodiment, blurring detection and image synthesis are 
performed by each small area AR(k) of the photo -sensing 
surface of the image sensing device IMS, therefore, the 
image synthesis using the equations (7) to (10) is also 
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performed by each small area AR(k) . Then, in a small 
area where blurring due to movement of an object occurs, 
the image is reproduced on the basis of one of the four 
images instead of synthesizing the four images. In 
this case, instead of the equations (7) to (10), the 
following equations are used. 



IMG(u=2i-l, v=2j) IGl(i, j) ...(37) 

IMG(u=2i-l, v=2j-l) <~ IGl(i, j) ...(38) 

IMG(u=2i, v=2j-l) <- IGl(i, j) ...(39) 

IMG(u=2i, v=2j) *- IGl(i, j) ...(40) 



More specifically, in an area of a synthesized 
image where blurring is not found and the image 
synthesis is performed on the basis of the equations 
(7) to (10), the quantity of information on the area as 
well as the number of pixels configuring the area is 
four times greater than each of the four images. 
Whereas, in an area of the synthesized image where 
blurring due to movement of the object occurred and a 
corresponding image portion is reproduced on the basis 
of the equations (37) to (40), the number of pixels 
configuring the area is also four times greater than 
each of the four images, however, the quantity of 
information is the same as that of each of the four 
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images . More specifically, the resolution of the image 
is the same as that of each of the four images. 

Note, the aforesaid image synthesis method is used 
when image signals are obtained from a black-and-white 
image sensing device and a multiple-CCD type color 
image sensing device using a color separation prism. 
When image signals are obtained from a single CCD type 
color image sensing device covered with a mosaic color 
filter, although there are little differences in pixel 
shifting amount in the pixel shifting operation and the 
image synthesis method, the basic ideas of the pixel 
shifting operation and the image synthesis are the same. 

Fig. 46 is a flowchart of determination processing 
on blurring due to vibration, as explained with 
reference to Figs. 26A to 32 and 9A to 10E, performed 
during pixel shifting operation. This processing 
corresponds to the process performed in step S126 in 
Fig. 40. 

First in step S161, a counter CNT for counting the 
number of image formation positions is initialized to 0. 

In step S162, correlation between the first and 
second image signals are calculated on the basis of the 
equation (30). The process proceeds to step S163 where 
the maximum shift amounts in the horizontal and 
vertical directions, STHmax and STVmax, are calculated 
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on the basis of the result of the correlation 
calculated in step S162. 

Thereafter, in step S164, the maximum correlation 
in the horizontal and vertical directions, CRHmax and 
CRVmax are calculated. Further, in step S165, the 
blurring amounts in the horizontal and vertical 
directions, 6 H and 6 V , are calculated. 

In step S166, the counter CNT is increased by 1, 
and in step S167, the value of the counter CNT and the 
predetermined value N SF indicating the number of image 
formation positions are compared. For example, if N SF 
is 4, calculations performed in steps S162 to S165 are 
performed three times between the first and second 
image signals, the second and third image signals, and 
the third and fourth image signals. Therefore, if CNT 
is less than N SF -1, then the process returns to step 
S162, and the processes in steps S162 to S166 are 
performed on the next pair of image signals. When the 
CNT reaches N SF -1, the process proceeds to step S168. 

In step S168, the respective blurring amounts 5 H 
and 6 V which are calculated in step SI 6 5 are compared 
to a predetermined value 6 X . If both of the blurring 
amounts 6 H and 8 V are less than the predetermined value 
5 1# it is determined that the blurring due to vibration 
is almost nil, and a flag FLHD for indicating whether 
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or not blurring due to vibration occurred (referred to 
as "vibration flag FLHD" hereinafter) is set to 0. 

Whereas, if it is determined in step S168 that at 
least one of the blurring amounts 6 H and 6 V is equal or 
greater than the predetermined value 8 1# then the 
process goes to step S174 where the vibration flag FLHD 
is set to 1, 

After a process in either step S173 or S174 is 
performed, the process proceeds to step S171 where the 
process returns to the processing in Fig. 40. 

Blurring due to vibration is determined as 
described above. 

Fig. 47 is a flowchart of blurring determination 
processing due to movement of the object occurred while 
performing pixel shifting operation, explained with 
reference to Figs. 43 and 44, and the blurring 
determination processing corresponds to step S127 in 
Fig. 40. 

First in step S1176, the photo-sensing surface of 
the image sensing device IMS is divided into the K 
number of small areas AR(k) as shown in Fig. 43. 

In step S1177, blurring amounts, 5^ and 6^ are 
calculated for each of the divided areas AR(k) . More 
specifically, the processes in steps S161 to S167 are 
repeatedly performed on the respective divided areas 
AR(k) , thereby blurring amounts in the horizontal and 
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vertical directions 5^ and 5^ are calculated for each 
of the divided areas AR(k) . 

In step S1178, the blurring amounts 5^ and 8^ 
calculated in step S1177 are compared to a 
predetermined value 6 2 . If both of the blurring amounts 
8^ and 6^ are smaller than the predetermined value 5 2 , 
then it is determined that the blurring due to movement 
of the object in the divided area is almost nil. 
Thereafter, the process proceeds to step S1179 where a 
flag FLOB(k) indicating occurrence of blurring due to 
movement of the object (referred to as "movement flag 
FLOB" hereinafter) is set to 0. 

Whereas, if at least one of the blurring amounts 
5^ and 5^ is greater than the predetermined value 6 2 , 
in step S1178, then the process proceeds to step S1180, 
where the movement flag FLOB(k) is set to 1. 

In step S1181, whether or not the processes in 
steps S1177 to S1180 have been performed on all the 
divided areas AR(k) is determined* If not, the process 
returns to step S1177 where the processes in steps 
S1177 to S1180 are performed on the unprocessed divided 
area. Whereas, if it is determined in step S1181 that 
the processes in steps S1177 to S1180 are performed on 
all the divided areas AR(k) , then the process proceeds 
to step SI 182 where the process returns to the 
processing shown in Fig. 40. 
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In the aforesaid manner, blurring due to movement 
of the object in each divided area is determined. 

Fig. 48 is a flowchart of the image synthesis 
processing explained with reference to Figs. 45 and 9A 
to 10E. This processing is a subroutine corresponding 
to the process in step S128 in Fig. 40. 

First in step S1186, whether or not there is 
blurring due to vibration between the plurality of 
images sensed in pixel shifting operation is determined. 
More specifically, the status of the vibration flag 
FLHD which is set either in step S173 or S174 in Fig. 
46 is checked. If the status of the vibration flag 
FLHD is 1, which shows that the blurring is large, then 
the process moves to step S1194 where a warning message 
is displayed. Thereafter, the process returns to the 
main processing in step S1195. 

If it is determined in step S1186 that the 
vibration flag FLHD is 0, namely, if it is determined 
that blurring due to vibration is small, then the 
process proceeds to step S1187. 

In step S1187, in order to determine blurring due 
to movement of the object for each small area AR(k) 
divided as shown in Fig. 43, an independent variable k 
is reset to 0. 

In step S1188, blurring due to movement of the 
object in each divided area AR(k) is evaluated. More 
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specifically, the status of the movement flag FLOB(k) 
set in either step S1179 or step S1180 in Fig. 47 is 
checked. If the movement flag FLOB(k) is 1, which 
indicates that blurring is large, then the process 
5 proceeds to step S1189, where an image in the 

corresponding divided area AR(k) is reproduced by using 
one of the plurality of images using the equations (37) 
to (40). 

E Whereas, if it is determined in step S1188 that 

10 the movement flag FLOB(k) is 0, which shows that the 
O blurring is small, then the process proceeds to step 

^ S1190, and image synthesis using the four images is 

^ performed using the equations (7) to (10). 

H= After the process in step S1189 or S1190 is 

fU 15 performed, the process proceeds to step S1191, where 
fy the independent variable k is increased by 1. 

In step S1192, whether or not the independent 
variable k reaches the number of division K of the 
divided areas AR(k) is determined. If k is less than K, 
20 then the process returns to step S1188 and 

determination of blurring due to movement of the object 
is repeatedly performed, and when k reaches K, the 
process proceeds to step S1193. 

In step S1193, the contents of the image synthesis 
25 mode performed in either step S1189 or S1190 is 

displayed on the display DISP to notify the user of 
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which image synthesis is performed. In step S1195, the 
process returns to the main processing shown in Fig. 40 

In the aforesaid processing, when the vibration is 
large, even though a plurality of images have been 
obtained by performing pixel shifting operation, image 
synthesis of the obtained images is not performed. 
Further, when the blurring due to movement of an object 
is large in a particular area, synthesis of a plurality 
of images is not performed for reproducing an image in 
the area , whereas , an area where blurring did not occur 
is reproduced by synthesizing the plurality of images 
for increasing resolution. Furthermore, if neither 
blurring due to vibration nor blurring due to movement 
of an object occurs in the entire image, then the 
entire image is reproduced by synthesizing the 
plurality of images in order to increase the resolution 

According to the eleventh embodiment as described 
above , 

(1) Blurring due to movement of an object is 
extracted on the basis of correlation between a 
plurality of images obtained while performing pixel 
shifting operation, and method for reproducing each of 
a plurality of divided areas of a final image is 
changed in accordance with the degree of blurring. 
Accordingly, it is possible to prevent an unnatural 
image, caused by movement of the object, from being 



-149- 



generated, as well as it is possible to obtain an image 
of high resolution as a result of pixel shifting 
operation* 

(2) Only the area or areas where blurring due to 
5 movement of an object are reproduced using one of a 

plurality of images, instead of synthesizing the 
plurality of images. Accordingly, although the 
resolution in the aforesaid area or areas of an image 
is not increased, an image portion in the area or areas 
10 is expressed as if it is stationary. At the same time, 
it is possible to reproduce the other area or areas in 
high resolution. 

(3) The type of image synthesis of a plurality of 
images which are obtained by performing pixel shifting 

15 operation is displayed on display means to notify the 
user, the user is able to check the resolution of the 
obtained image as well as how the moving object is 
reproduced. Accordingly, it is possible for the user 
to designate to resense an image, for example, if an 

20 undesired image is obtained. 

(4) Since the camera has an optical image 
stabilization means, it is possible to reduce 
deterioration of an image due to vibration when 
performing pixel shifting operation. 



25 
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In the eleventh embodiment, the area where 
blurring due to movement of an object occurred is 
reproduced using one of a plurality of images instead 
of synthesizing the plurality of images, thereby 
5 expressing the area as if an object in the area is not 
moving. In the twelfth embodiment, process to connect 
between the plurality of images is performed in the 
area where blurring due to movement of the object, 
thereby expressing the movement of the object. 
10 Figs. 49 to 51 are views for explaining image 

synthesis method according to the twelfth embodiment, 
and Fig. 52 is a flowchart of control processing 
performed by the microcomputer CCPU of the camera main 
body CMR. Below, the twelfth embodiment will be 
15 explained with reference to Figs. 49 to 52. 

As explained with reference to Fig. 41, when the 
object whose part is moving is sensed while performing 
pixel shifting operation, images each of which can be 
basically classified into two different types of areas 
20 are obtained: the area of one type shows the moving 
portion of the object and large shift is observed in 
the area between the plurality of images; and the area 
of the other shows a still image and slight shift due 
to pixel shifting operation is observed in this area. 
25 Correlation between a first image and a second image is 
calculated, and an area where blurring due to movement 
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of the object is determined and extracted. The 
determined area is shown as an area ARMV surrounded by 
a bold line in Fig. 49. 

Next, correlation between images within the area 
5 ARMV is calculated again, and an area corresponding to 
the moving portion of the object is extracted. As a 
result, it is determined that the moving portion in the 
first image is in the area OBMV x , and the moving 
p portion of the second image is the area OBMV 2 . 

Iff 10 Thereafter, movement vectors between these areas OBMV r 

yQ and OBMV 2 of the first and second images are determined. 

When calculating correlation between the two images at 
this time, the shifted amounts between the two images 
not only in the linear directions but also in the 
15 rotational direction are taken into consideration to 
calculate correlation. Accordingly, movement between 
the first and second images, including angular 
component, is determined. Referring to Fig. 49, it is 
determined that the moving portion in the images is 
20 rotating by an angle 6^ about a point G^. 

Fig. 50 is a view for explaining sweeping 
processing between the moving portions of the two 
images on the basis of the movement vectors. Referring 
to Fig. 49, a reference line of the first moving area 
25 OBMVi at the start points of the movement vectors is 

extracted. The extracted reference line is shown as a 
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line SHi (first reference line) in Fig. 50. Next, a 
reference line of the second moving area OBMV 2 at the 
start points of the movement vectors is extracted, and 
this reference line is expressed by a line SH 2 (second 
reference line) in Fig. 50. Then the first reference 
line SHi is swept toward the second reference line SH 2 
in accordance with the movement vectors while 
generating image signals between the first and second 
reference line lines SI^ and SH 2 . When pixels between 
the first and second reference line lines SH X and SH 2 
are filled, then the sweeping processing is finished. 

The aforesaid sweeping processing is also 
performed between the second and third images, and 
between the third and fourth images. Further, an image 
of the moving portion in the fourth image (MVOB 4 ) is 
used for reproducing the final image. Meanwhile, 
images in areas where blurring due to movement of the 
object is not detected are reproduced by performing a 
normal image synthesis suitable for the performed pixel 
shifting operation. 

Fig. 51 shows an example of the final image 
obtained by performing the aforesaid image synthesis 
processing. The area where blurring due to movement of 
an object is determined is expressed by a swept area 
ARSP and the moving portion OBMV 4 of the fourth image. 
In this manner, while expressing the movement of the 
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object, the arm of the person is expressed as a clear 
image. Further, by using sweeping expression, it is 
possible to prevent the background image from being 
seen through the object. 

Fig. 52 is a flowchart of control processing 
performed by the microcomputer CCPU of the camera main 
body CMR according to the twelfth embodiment of the 
present invention. Control processing of the camera 
main body CMR, processing for determining blurring due 
to vibration, processing for determining blurring due 
to movement of an object, and control processing of the 
lens LNS are the same as those explained in the 
eleventh and first embodiments with reference to Figs. 
40, 46, 47, and 7. A process in the twelfth embodiment 
which is different from the eleventh embodiment is the 
process in step S124 in Fig. 40 for synthesizing images. 
The processing performed in step S124, according to the 
twelfth embodiment, is shown in Fig. 52. 

Referring to Fig. 52, first in step S1286, whether 
or not there is blurring due to vibration between the 
plurality of images sensed in pixel shifting operation 
is determined. More specifically, similarly to the 
eleventh embodiment, the status of the vibration flag 
FLHD which is set either in step S173 or S174 in Fig. 
46 is checked. If the status of the vibration flag 
FLHD is 1, which shows that the blurring is large, then 
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the process moves to step S1294 where a warning message 
indicative of image synthesis not being performed is 
displayed. Thereafter, the process returns to the main 
processing in step S1297. 

If it is determined in step S1286 that the 
vibration flag FLHD is 0, namely, if it is determined 
that blurring due to vibration is small, then the 
process proceeds to step S1287. 

In step S1287, where or not blurring due to 
movement of the object occurred is determined. More 
specifically, the statuses of the movement flags 
FLOB(k) set in either step S1179 or step S1180 in Fig. 
47 for respective divided areas AR(k) are checked, 
thereby determining whether or not there is any area 
where blurring due to movement of the object occurred. 
If it is determined that there is no area where 
blurring due to movement of the object occurred, then 
the process goes to step S1295, where the four images 
obtained in pixel shifting operation are synthesized 
using the equations (7) to (10), thereby obtaining an 
image expressed in high resolution. 

Thereafter, in step S1296, a message indicating 
that an image expressed in high resolution is obtained, 
or, "still image synthesis" is performed, is displayed 
on the display DISP. Then, the process proceeds to 
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step S1297 where the process returns to the main 
processing. 

Whereas, if it is determined in step S1287 that at 
least one divided area AR(k) where blurring due to 
movement of the object occurred, then the process 
proceeds to step S1288. 

In step S1288, the moving portion of the object is 
extracted as described with reference to Fig. 49. 

Thereafter in step S1289, movement vectors of the 
moving portion are calculated. 

In step S1290, a reference line of the moving 
portion is extracted as described with reference to Fig. 
50. 

In step S1291, sweeping processing is performed on 
the moving portion in the area where blurring occurs on 
the basis the movement vectors. Further, at the end 
position of the sweeping, the moving portion of the 
fourth image is displayed. 

Regarding the area where no blurring is detected, 
image synthesis using the equations (7) to (10) is 
performed in step S1292. 

In step S1293, a message indicating that sweeping 
processing is performed on the area where blurring due 
to movement of the object occurred, or, "moving image 
synthesis" is performed, is displayed on the display 
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DISP. Then, in step S1297, the process returns to the 
main processing. 

According to the aforesaid image synthesis method, 
an area where large blurring due to movement of an 
object occurred is extracted, and the area is applied 
with sweeping processing. Accordingly, it is possible 
to express motion of a moving portion of the object 
while reproducing an area where no blurring due to 
movement of object did not occur in high resolution by 
synthesizing a plurality of images. 

According to the twelfth embodiment as described 
above, in addition to the advantages (1), (3) and (4) 
of the eleventh embodiment, 

(5) Since motion of a moving portion of a 
plurality of images, obtained while performing pixel 
shifting operation, is reproduced by performing 
sweeping processing as well as the moving portion is 
displayed as a clear image using the last image of the 
plurality of images, it is possible to naturally 
express a moving object, such as a moving person and a 
moving car, as well as a non-moving portion of the 
plurality of images is reproduced as a high-precision 
image by synthesizing the plurality of images. 

<Thirteenth Embodiment > 
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In the twelfth embodiment, the gap between moving 
portions of an object in the plurality of images is 
filled by performing sweeping processing, further, the 
moving portion itself is displayed as a clear image 
using an image among the plurality of images which is 
sensed last. Accordingly, while the clear image of the 
moving portion is displayed, motion of the moving 
portion is expressed. In the thirteenth embodiment as 
described below, the moving portion is completely 
expressed by sweep lines, which are generated while 
performing sweeping processing. This method is 
suitable for synthesizing images obtained by so-called 
follow shot. 

Figs. 53 to 54 are views for explaining image 
synthesis method according to the thirteenth embodiment, 
and Fig. 55 is a flowchart of control processing 
performed by the microcomputer CCPU of the camera main 
body CMR. Below, the thirteenth embodiment will be 
explained with reference to Figs. 53 to 55. 

Fig. 53 corresponds to Fig. 50 explained in the 
twelfth embodiment, and shows a result of reference 
line extraction of a moving portion of the object which 
is extracted as described with reference to Fig. 49. 
In the twelfth embodiment, the reference line of the 
moving portion only at start points of movement vectors 
is extracted; whereas, in the thirteenth embodiment, 
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reference lines of the moving portion both at start 
points of the movement vectors and the side of 
destination points of the movement vectors are 
extracted. More specifically, referring to Fig. 53, 
when performing sweeping processing on first and second 
images, four reference lines, namely, a reference line 
SHu of the moving portion in the first image at start 
points of the movement vectors, a reference line SH 12 
of the moving portion in the first image at destination 
points of the movement vectors, a reference line SH 21 
of the moving portion in the second image at start 
points of the movement vectors, and a reference line 
SH 22 of the moving portion in the second image at 
destination points of the movement vectors, are 
extracted. 

Thereafter, the gap between the reference lines 
SH 1X and SH 21 is filled by performing sweeping 
processing as shown by solid arrows in Fig. 53, thus an 
image in the area between the reference lines SH 1:L and 
SH 21 is generated. Likewise, the gap between the 
reference lines SH 12 and SH 22 is filled by performing 
sweeping processing as shown by broken arrows, thus an 
image in the area between the reference lines SH 12 and 
SH 22 is generated. The aforesaid sweeping processing is 
performed on the second and third images, and the third 
and fourth images. Thereafter, the image of the area 
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between the reference lines SH 1X and SH 21 and the image 
of the area between the reference lines SH 12 and SH 22 
are combined by averaging signal values of the images . 
Accordingly, the moving portion of the object is 
expressed as if it is continuously moving in the 
direction of movement vector throughout the area where 
blurring due to movement of the object occurred. As 
for the area where blurring due to movement of the 
object did not occur, similarly to the twelfth 
embodiment, a normal image synthesis suitable for the 
performed pixel shifting operation is performed to 
generate a high-resolution image. 

Fig. 54 shows a final image obtained by performing 
image synthesis operation according to the thirteenth 
embodiment. The area where blurring due to movement of 
the object occurred is expressed by a swept area ARSP. 
In this area, the moving portion of the object is 
expressed as if it is continuously moving, which 
further enhances movement of the object comparing to 
the twelfth embodiment. In addition, by using sweeping 
expression, it is possible to prevent the background 
image from being seen through the object. 

Fig. 55 is a flowchart of control processing 
performed by the microcomputer CCPU of the camera main 
body CMR according to the thirteenth embodiment of the 
present invention. The control processing of the 
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thirteenth embodiment is almost the same as that shown 
in Fig, 52 explained in the twelfth embodiment except 
processes in step S1390 to S1392, therefore, these 
processes are only explained below. 

The process in step S1290 for extracting reference 
lines and the process in step S1291 for performing 
sweeping processing in the image synthesis processing 
shown in Fig. 52 explained in the twelfth embodiment 
are replaced by a process in step S1390 for extracting 
reference lines and a process in step SI 391 for 
performing sweeping processing according to the 
thirteenth embodiment in the image synthesis processing 
shown in Fig. 55, further, a process in step S1392 for 
averaging signal values of images is added. Details 
follow. 

In the process in step S1390 for extracting 
reference lines, reference lines of the moving portion 
at both the start and destination points of the 
movement vectors are extracted. 

In step S1391, the sweeping processing is 
performed on the areas between the reference lines at 
the start points of the movement vectors and between 
the reference lines at the destination points of the 
movement vectors. 

Then, in step S1392, the two image portions, 
obtained by performing the sweeping processing, between 
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the two sets of reference lines are combined by- 
averaging signal values of the two image portions. 

Note, the processes in steps S1390 to S1392 are 
performed on every pair of successive images. 
5 According to the aforesaid image synthesis method, 

an area where large blurring due to movement of an 
object occurred is extracted, and the area is applied 
with sweeping processing. Accordingly, it is possible 
to express motion of a moving portion of the object 

10 while reproducing an area where no blurring due to 

movement of object did not occur in high resolution by 
synthesizing a plurality of images. 

According to the thirteenth embodiment as 
described above, in addition to the advantages (1), (3) 

15 and (4) of the eleventh embodiment, 

(6) Since motion of a moving portion in a 
plurality of images, obtained while performing pixel 
shifting operation, is reproduced by performing 
sweeping processing, it is possible to naturally 

20 express a moving object, such as a moving person and a 
moving car, as well as a non-moving portion of the 
plurality of images is reproduced as a high-precision 
image by synthesizing the plurality of images. 
Especially, the image synthesis method of the 

25 thirteenth embodiment is suitable when the plurality of 
images are obtained in so-called follow shot, a method 
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for sensing a moving object by following it while 
letting the background flow. 

<Fourteenth Embodiment> 
5 In the eleventh to thirteenth embodiments, the 

camera automatically selects image synthesis method in 
accordance with the states of blurring in images due to 
vibration and movement of an object. In the fourteenth 
embodiment, after the camera senses a plurality of 
10 images by performing pixel shifting operation, the user 
selects a desired image synthesis method. 

In the fourteenth embodiment, a predetermined 
program is selected using an image synthesis mode 
selection switch SWCMP and executed in response to an 
15 operation of an image synthesis start switch SWST, 
where both the switches are shown in Fig. 1. 

The image synthesis mode selection switch SWCMP 
switches between a plurality of positions connected to 
the microcomputer CCPU, e.g., four positions, and the 
20 user sets the switch SWCMP to one of the positions so 
as to select a desired image synthesis mode. 

The image synthesis start switch SWST is a 
pushbotton switch which is also connected to the 
microcomputer CCPU, and when the switch SWST is pressed 
25 by the user, image synthesis conforming to the image 
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synthesis mode selected by the image synthesis 
selection switch SWCMP starts. 

Fig. 56 is a flowchart of control processing 
performed by the microcomputer CCPU of the camera main 
body CMR according to the fourteenth embodiment of the 
present invention. The processing according to the 
fourteenth embodiment is explained below with reference 
to Fig. 56. 

After images of an object is sensed while 
performing pixel shifting operation, as explained in 
the aforesaid embodiments, the image synthesis start 
switch SWST is turned on. In turn, in step S1401, the 
image synthesis starts. 

In step S1402, the status of the image synthesis 
mode selection switch SWCMP is determined, thereby the 
selected image synthesis mode is recognized. 

Thereafter, in step S1403, the selected image 
synthesis mode is displayed on the display DISP. 

In step S1404, the image synthesis processing 
corresponding to the selected image synthesis mode is 
determined. If it is determined in step S1404 that a 
mode which does not perform image synthesis of the 
plurality of images (indicated by the number "0") is 
selected, then the image synthesis is not performed, 
and the process proceeds to step S1408 where the 
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respective images or one of the images is stored 
without being processed in the memory MEM. 

If it is determined in step S1404 that "stop- 
motion mode" (indicated by the number "1") is selected, 
5 then the process proceeds to step S1405 where the 

processing explained in the eleventh embodiment with 
reference to Figs. 47 and 48 is performed, thereby an 
image as shown in Fig. 45 is reproduced. Thereafter, 
in step S1408, the synthesized image is stored in the 

10 memory MEM. 

If it is determined in step S1404 that "moving 
mode" (indicated by the number "2") is selected, then 
the process proceeds to step S1406 where the processing 
explained in the twelfth embodiment with reference to 

15 Fig. 52 is performed, thereby an image as shown in Fig. 
51 is reproduced. Thereafter, in step S1408, the 
synthesized image is stored in the memory MEM. 

If it is determined in step S1404 that "follow 
shot mode" (indicated by the number "3") is selected, 

20 then the process proceeds to step S1407 where the 

processing explained in the thirteenth embodiment with 
reference to Fig. 55 is performed, thereby an image as 
shown in Fig. 54 is reproduced. Thereafter, in step 
S1408, the synthesized image is stored in the memory 

25 MEM. 
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As described above, image synthesis corresponding 
to the selected image synthesis mode is performed, and 
the obtained image is stored. Thereafter, the obtained 
image signals are transmitted to an external computer 
or a printer via the connector CNC in step S1409. In 
step S1410, the image synthesis and output processes 
are completed. 

According to the fourteenth embodiment as 
described above, 

(7) After a plurality of images are obtained 
while performing pixel shifting operation, a 
synthesized image is obtained by performing one of 
different image synthesis methods in response to an 
image synthesis instruction operation which is 
performed independently of the image sensing processing. 
Accordingly, image synthesis mode is selected after the 
plurality of images are obtained; therefore, it is 
unnecessary to select image synthesis mode to be used 
when taking a photograph. Thus, possibility to lose a 
shutter change is reduced, and a desired image 
synthesis method can be selected after finishing image 
sensing operation. 

(8) By repeating the image synthesis operation by 
using different image synthesis modes, it is possible 
to obtain differently expressed images using the same 
set of images. Thus, it is possible to save memory 
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area for storing images as well as the sensed images 
can be used for generating images of different 
expressions. 

<Modif ications> 

The above eleventh to fourteenth embodiments can 
be applied to an image sensing apparatus which senses a 
plurality of images in a predetermined period and 
synthesizes the sensed images for purposes other than 
pixel shifting operation. For example, by applying the 
eleventh to fourteenth embodiments to an image sensing 
apparatus which senses a plurality of images while 
changing luminous exposure and synthesizes the 
plurality of images in order to widen the dynamic range 
of a synthesized image, it is possible to obtain an 
image of a wide dynamic range without being affected by 
vibration. 

Further, in a case where an interline scanning 
type CCD, which has been developed for sensing a moving 
image, is used in the image sensing apparatus, a frame 
image of high resolution is obtained by combining two 
field images sensed at different times. When the 
eleventh to fourteenth embodiments are applied to such 
the image sensing apparatus, it is possible to prevent 
blurring due to movement of the object from occurring 
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in the two field images, therefore, it is possible to 
obtain a high-quality image. 

The present invention is not limited to the above 
embodiments and various changes and modifications can 
be made within the spirit and scope of the present 
invention. Therefore to apprise the public of the 
scope of the present invention, the following claims 
are made. 
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